[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:28 dist-test-slave-rvjj krb5kdc[18618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:28 dist-test-slave-rvjj krb5kdc[18618](info): set up 2 sockets
Sep 23 18:05:28 dist-test-slave-rvjj krb5kdc[18618](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 18:05:29.472597 18611 mini_kdc.cc:121] Time spent starting KDC: real 2.308s user 0.004s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:29 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:29.630082 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46473
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:29.757369 18631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:29.757676 18631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:29.757735 18631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:29.762637 18631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:29.762758 18631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:29.762789 18631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:29.762812 18631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:29.762835 18631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:29.768882 18631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:29.770108 18631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:29.771330 18631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:29.781663 18637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:29.781731 18631 server_base.cc:1034] running on GCE node
W20260923 18:05:29.781668 18636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:29.781663 18639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:29.782708 18631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:29.784099 18631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:29.785290 18631 hybrid_clock.cc:648] HybridClock initialized: now 1790186729785258 us; error 51 us; skew 500 ppm
Sep 23 18:05:29 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186729, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:29.788728 18631 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:29.789892 18631 webserver.cc:460] Webserver started at http://127.18.44.254:43939/ using document root <none> and password file <none>
I20260923 18:05:29.790567 18631 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:29.790650 18631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:29.790895 18631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:29.793322 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "2deaa12c78384bceaa38b554dc53d5d9"
format_stamp: "Formatted at 2026-09-23 18:05:29 on dist-test-slave-rvjj"
server_key: "48f9cb45260d7f4191a449ed5c5eb561"
server_key_iv: "da73982b112f9e9cd2fdf1d49e27adf6"
server_key_version: "encryptionkey@0"
I20260923 18:05:29.793991 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "2deaa12c78384bceaa38b554dc53d5d9"
format_stamp: "Formatted at 2026-09-23 18:05:29 on dist-test-slave-rvjj"
server_key: "48f9cb45260d7f4191a449ed5c5eb561"
server_key_iv: "da73982b112f9e9cd2fdf1d49e27adf6"
server_key_version: "encryptionkey@0"
I20260923 18:05:29.798449 18631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:05:29.802094 18646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:29.804001 18631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 18:05:29.804208 18631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "2deaa12c78384bceaa38b554dc53d5d9"
format_stamp: "Formatted at 2026-09-23 18:05:29 on dist-test-slave-rvjj"
server_key: "48f9cb45260d7f4191a449ed5c5eb561"
server_key_iv: "da73982b112f9e9cd2fdf1d49e27adf6"
server_key_version: "encryptionkey@0"
I20260923 18:05:29.804356 18631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:29.815372 18631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:29.820021 18631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:29.820262 18631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:29.828544 18631 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:46473
I20260923 18:05:29.828562 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:46473 every 8 connection(s)
I20260923 18:05:29.829634 18631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:29.832891 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:29.838295 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 18631
I20260923 18:05:29.838521 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:29.840045 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9: Bootstrap starting.
I20260923 18:05:29.840368 18611 external_mini_cluster.cc:1428] Setting key 62d3e16f0c27556bbb8e63c776749f4b
I20260923 18:05:29.843696 18699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:29.844538 18699 log.cc:824] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:29.847169 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9: No bootstrap required, opened a new log
I20260923 18:05:29.851773 18699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } }
I20260923 18:05:29.852145 18699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:29.852212 18699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2deaa12c78384bceaa38b554dc53d5d9, State: Initialized, Role: FOLLOWER
I20260923 18:05:29.852764 18699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } }
I20260923 18:05:29.852919 18699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:29.852984 18699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:29.853096 18699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:29.854200 18699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } }
I20260923 18:05:29.854595 18699 leader_election.cc:302] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2deaa12c78384bceaa38b554dc53d5d9; no voters:
I20260923 18:05:29.854936 18699 leader_election.cc:288] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:29.855069 18704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:29.855352 18704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 2deaa12c78384bceaa38b554dc53d5d9, State: Running, Role: LEADER
I20260923 18:05:29.855753 18704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } }
I20260923 18:05:29.856283 18699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:29.857437 18706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2deaa12c78384bceaa38b554dc53d5d9. Latest consensus state: current_term: 1 leader_uuid: "2deaa12c78384bceaa38b554dc53d5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } } }
I20260923 18:05:29.857594 18706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:29.858561 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2deaa12c78384bceaa38b554dc53d5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deaa12c78384bceaa38b554dc53d5d9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46473 } } }
I20260923 18:05:29.858702 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:29.859576 18713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:29.863152 18713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 18:05:29 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:29.870369 18713 catalog_manager.cc:1347] Generated new cluster ID: c5d75165604e409b93e765a5565389a5
I20260923 18:05:29.870452 18713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:29.873555 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:29.845327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.18.44.254:46473)
0923 18:05:29.845886 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:29.845907 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:29.846323 (+ 416us) server_negotiation.cc:407] Connection header received
0923 18:05:29.851130 (+ 4807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:29.851166 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:29.851733 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:29.852215 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:29.855369 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:29.856948 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:29.859353 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:29.860257 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:29.860435 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:29.866447 (+ 6012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:29.866454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:29.866468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:29.866510 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:29.868694 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:29.869535 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:29.869543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:29.869549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:29.869748 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:29.870261 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:29.870264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:29.870266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:29.870929 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:29.871349 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:29.872136 (+ 787us) server_negotiation.cc:299] Negotiation successful
0923 18:05:29.872472 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":99,"threads_started":1}
I20260923 18:05:29.887662 18713 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:29.889078 18713 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:29.896001 18713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2deaa12c78384bceaa38b554dc53d5d9: Generated new TSK 0
I20260923 18:05:29.897020 18713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:29.940722 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:30.076864 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:30.077205 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:30.077255 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:30.082010 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:30.082118 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:30.082314 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:30.088837 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:30.090081 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:30.091289 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:05:30.102767 18725 server_base.cc:1034] running on GCE node
W20260923 18:05:30.102878 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.102793 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.102748 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:30.103519 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:30.104277 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:30.105489 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790186730105448 us; error 58 us; skew 500 ppm
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:30.108986 18725 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:30.110049 18725 webserver.cc:460] Webserver started at http://127.18.44.193:37487/ using document root <none> and password file <none>
I20260923 18:05:30.110736 18725 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:30.110818 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:30.111037 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:30.113408 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "a56ba43d632c4720887c9dfc1173c2e8"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "8d457cf1e6a265a3b50b1811f8982865"
server_key_iv: "28bf5f0415f00f27a7faa5f62d287243"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.114055 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "a56ba43d632c4720887c9dfc1173c2e8"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "8d457cf1e6a265a3b50b1811f8982865"
server_key_iv: "28bf5f0415f00f27a7faa5f62d287243"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.119135 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 18:05:30.122363 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.124114 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:30.124279 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "a56ba43d632c4720887c9dfc1173c2e8"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "8d457cf1e6a265a3b50b1811f8982865"
server_key_iv: "28bf5f0415f00f27a7faa5f62d287243"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.124433 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:30.155133 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:30.158948 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:30.159127 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:30.159799 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:30.160661 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:30.160707 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.160753 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:30.160773 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.169798 18725 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:34329
I20260923 18:05:30.170068 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:34329 every 8 connection(s)
I20260923 18:05:30.170866 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:30.179754 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 18725
I20260923 18:05:30.179884 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:30.180174 18611 external_mini_cluster.cc:1428] Setting key a76f56dbcc884f899f21323bd2b2024f
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:30.186796 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.173106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:59443 (local address 127.18.44.254:46473)
0923 18:05:30.173284 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:30.173290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:30.174613 (+ 1323us) server_negotiation.cc:407] Connection header received
0923 18:05:30.175644 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:30.175647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:30.175713 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:30.175888 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:30.177377 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.178013 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.179432 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.179783 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.179993 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.183124 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:30.183127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:30.183129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:30.183157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:30.184601 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.185170 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.185173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.185174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.185258 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.185613 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.185617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.185619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.185816 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:30.185904 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:30.186471 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 18:05:30.186622 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:05:30.187739 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.173582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46473 (local address 127.18.44.193:59443)
0923 18:05:30.174411 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:30.174464 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:30.175394 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:30.176075 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:30.176084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:30.176511 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:30.177082 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.177098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.178194 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.178198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.179223 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.179236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.180082 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.180086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.180316 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.180981 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:30.181001 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:30.182923 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:30.184741 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.184748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.184768 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.185064 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.185352 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.185355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.185357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.185522 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.185925 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:30.185933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:30.186220 (+ 287us) client_negotiation.cc:769] Sending connection context
0923 18:05:30.186411 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 18:05:30.186776 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:30.189126 18854 heartbeater.cc:347] Connected to a master server at 127.18.44.254:46473
I20260923 18:05:30.189428 18854 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:30.189975 18854 heartbeater.cc:511] Master 127.18.44.254:46473 requested a full tablet report, sending...
I20260923 18:05:30.191768 18663 ts_manager.cc:194] Registered new tserver with Master: a56ba43d632c4720887c9dfc1173c2e8 (127.18.44.193:34329)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:30.193094 18663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:59443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:30.213106 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:30.337129 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:30.337455 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:30.337500 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:30.341962 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:30.342056 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:30.342123 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:30.348450 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:30.349757 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:30.350962 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:30.362452 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.362452 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.362670 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:30.362861 18859 server_base.cc:1034] running on GCE node
I20260923 18:05:30.363703 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:30.364494 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:30.365731 18859 hybrid_clock.cc:648] HybridClock initialized: now 1790186730365689 us; error 68 us; skew 500 ppm
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:30.369306 18859 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:30.370513 18859 webserver.cc:460] Webserver started at http://127.18.44.194:35433/ using document root <none> and password file <none>
I20260923 18:05:30.371167 18859 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:30.371248 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:30.371501 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:30.373756 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "9b4101839e4041bb964ff36be4775172"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "7ecf60903e4159b9c3039810d1173da6"
server_key_iv: "3e078cdcae667265bcc86f41763966eb"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.374406 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "9b4101839e4041bb964ff36be4775172"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "7ecf60903e4159b9c3039810d1173da6"
server_key_iv: "3e078cdcae667265bcc86f41763966eb"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.379024 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:05:30.382414 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.384486 18859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:30.384696 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "9b4101839e4041bb964ff36be4775172"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "7ecf60903e4159b9c3039810d1173da6"
server_key_iv: "3e078cdcae667265bcc86f41763966eb"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.384861 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:30.413645 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:30.418551 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:30.418783 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:30.419499 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:30.420482 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:30.420537 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.420593 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:30.420622 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.431915 18859 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:35529
I20260923 18:05:30.431988 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:35529 every 8 connection(s)
I20260923 18:05:30.433238 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:30.441841 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 18859
I20260923 18:05:30.441967 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:30.442211 18611 external_mini_cluster.cc:1428] Setting key 54e54aba146b7393e929b23afb3d178c
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:30.454207 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.436242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:39123 (local address 127.18.44.254:46473)
0923 18:05:30.436538 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:30.436545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:30.437760 (+ 1215us) server_negotiation.cc:407] Connection header received
0923 18:05:30.439601 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:30.439605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:30.439692 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:30.439779 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:30.441620 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.442303 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.444155 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.444731 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.444863 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.448808 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:30.448814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:30.448818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:30.448868 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:30.450955 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.451853 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.451859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.451861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.452045 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.452543 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.452578 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.452580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.452928 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:30.453069 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:30.453769 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 18:05:30.453978 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:05:30.455144 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.436642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46473 (local address 127.18.44.194:39123)
0923 18:05:30.437503 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:30.437577 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:30.439201 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:30.440065 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:30.440076 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:30.440640 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:30.441347 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.441367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.442555 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.442561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.443908 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.443929 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.444991 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.444997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.445342 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.446182 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:30.446206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:30.448396 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:30.451194 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.451202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.451229 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.451670 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.452202 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.452205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.452207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.452406 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.453081 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:30.453088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:30.453439 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 18:05:30.453636 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 18:05:30.454043 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":166,"threads_started":1}
I20260923 18:05:30.456811 18988 heartbeater.cc:347] Connected to a master server at 127.18.44.254:46473
I20260923 18:05:30.457279 18988 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:30.458348 18988 heartbeater.cc:511] Master 127.18.44.254:46473 requested a full tablet report, sending...
I20260923 18:05:30.460235 18663 ts_manager.cc:194] Registered new tserver with Master: 9b4101839e4041bb964ff36be4775172 (127.18.44.194:35529)
I20260923 18:05:30.461206 18663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:39123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:30.480685 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:30.613956 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:30.614291 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:30.614338 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:30.618721 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:30.618811 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:30.618875 18993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:30.625226 18993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:46473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:30.626431 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:30.627790 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:30.640673 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.640709 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:30.640673 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:30.641454 18993 server_base.cc:1034] running on GCE node
I20260923 18:05:30.641978 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:30.642973 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:30.644397 18993 hybrid_clock.cc:648] HybridClock initialized: now 1790186730644368 us; error 56 us; skew 500 ppm
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:30.648572 18993 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:30.649863 18993 webserver.cc:460] Webserver started at http://127.18.44.195:45003/ using document root <none> and password file <none>
I20260923 18:05:30.650493 18993 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:30.650568 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:30.650807 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:30.653692 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "2a3d2f95c8204712968a5ac63ef50d36"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "d521551be1f31f1685a12f0abe009fd5"
server_key_iv: "ab5b3df9101622eb95c9914d7185ef7b"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.654807 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "2a3d2f95c8204712968a5ac63ef50d36"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "d521551be1f31f1685a12f0abe009fd5"
server_key_iv: "ab5b3df9101622eb95c9914d7185ef7b"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.661458 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 18:05:30.665748 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.668254 18993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:05:30.668452 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "2a3d2f95c8204712968a5ac63ef50d36"
format_stamp: "Formatted at 2026-09-23 18:05:30 on dist-test-slave-rvjj"
server_key: "d521551be1f31f1685a12f0abe009fd5"
server_key_iv: "ab5b3df9101622eb95c9914d7185ef7b"
server_key_version: "encryptionkey@0"
I20260923 18:05:30.668686 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:30.689890 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:30.694854 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:30.695080 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:30.695814 18993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:30.696867 18993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:30.696947 18993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.697008 18993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:30.697039 18993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:30.709674 18993 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46329
I20260923 18:05:30.709692 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46329 every 8 connection(s)
I20260923 18:05:30.710767 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:30.711016 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 18993
I20260923 18:05:30.711158 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:30.711556 18611 external_mini_cluster.cc:1428] Setting key ff0b7f31cbd9353caf8b0520942ab5ff
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:30.738494 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.714173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:36979 (local address 127.18.44.254:46473)
0923 18:05:30.714360 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:30.714365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:30.717331 (+ 2966us) server_negotiation.cc:407] Connection header received
0923 18:05:30.718852 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:30.718856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:30.719042 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:30.719188 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:30.721463 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.722224 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.725434 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.725787 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.725903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.731906 (+ 6003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:30.731909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:30.731912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:30.731953 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:30.734250 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.735841 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.735846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.735848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.735995 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.736677 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.736681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.736683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.736970 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:30.737083 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:30.738147 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0923 18:05:30.738323 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:05:30.740091 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.715133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46473 (local address 127.18.44.195:36979)
0923 18:05:30.716857 (+ 1724us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:30.716956 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:30.718437 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:30.719566 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:30.719578 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:30.720160 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:30.720927 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.720953 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.722609 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.722615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.725076 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:30.725104 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.726029 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.726034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:30.726271 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.728705 (+ 2434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:30.728733 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:30.731564 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:30.734531 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.734541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.734570 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.735407 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.736163 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:30.736166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:30.736168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:30.736446 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:30.737213 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:30.737223 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:30.737703 (+ 480us) client_negotiation.cc:769] Sending connection context
0923 18:05:30.737988 (+ 285us) client_negotiation.cc:240] Negotiation successful
0923 18:05:30.738468 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":742,"threads_started":1}
I20260923 18:05:30.743512 19122 heartbeater.cc:347] Connected to a master server at 127.18.44.254:46473
I20260923 18:05:30.744155 19122 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:30.746325 19122 heartbeater.cc:511] Master 127.18.44.254:46473 requested a full tablet report, sending...
I20260923 18:05:30.748736 18663 ts_manager.cc:194] Registered new tserver with Master: 2a3d2f95c8204712968a5ac63ef50d36 (127.18.44.195:46329)
I20260923 18:05:30.749572 18663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:36979
I20260923 18:05:30.759850 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:30.796473 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.781134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.18.44.193:34329)
0923 18:05:30.781474 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:30.781482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:30.781517 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:05:30.781925 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:30.781932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:30.782177 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:30.782342 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:30.783559 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.784546 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.785845 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.786332 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.786467 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.790347 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:30.790369 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:30.790391 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:30.790446 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:30.792823 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.793747 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.793759 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.793761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.794044 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.794763 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.794767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.794769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.795327 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:30.795845 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:30.796007 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 18:05:30.796248 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":127,"threads_started":1}
W20260923 18:05:30.798210 18768 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39662
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[18618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:30.837146 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:30.821671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.18.44.193:34329)
0923 18:05:30.822077 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:30.822083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:30.822235 (+ 152us) server_negotiation.cc:407] Connection header received
0923 18:05:30.822430 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:30.822433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:30.822494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:30.822576 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:30.824621 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.825765 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.826917 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:30.827353 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:30.827491 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:30.832290 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:30.832294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:30.832297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:30.832348 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:30.834283 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.835172 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.835177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.835179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.835361 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:30.835821 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:30.835824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:30.835826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:30.836181 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:30.836339 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:30.836670 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 18:05:30.836940 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260923 18:05:30.838923 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 18725
I20260923 18:05:30.846131 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 18859
I20260923 18:05:30.855350 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 18993
I20260923 18:05:30.862432 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 18631
2026-09-23T18:05:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3737 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[19136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[19136](info): set up 2 sockets
Sep 23 18:05:30 dist-test-slave-rvjj krb5kdc[19136](info): commencing operation
krb5kdc: starting...
W20260923 18:05:31.884361 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.010s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:31 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:32.048979 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33885
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:32.175962 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:32.176288 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:32.176333 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:32.180874 19149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:32.181084 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:32.181110 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:32.181129 19149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:32.181147 19149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:32.187428 19149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:33885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:32.188663 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:32.189805 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:05:32.200285 19149 server_base.cc:1034] running on GCE node
W20260923 18:05:32.200210 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.200237 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.200210 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:32.201759 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:32.203577 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:32.204856 19149 hybrid_clock.cc:648] HybridClock initialized: now 1790186732204814 us; error 70 us; skew 500 ppm
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186732, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:32.208878 19149 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:32.210256 19149 webserver.cc:460] Webserver started at http://127.18.44.254:36641/ using document root <none> and password file <none>
I20260923 18:05:32.211028 19149 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:32.211115 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:32.211408 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:32.214466 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "bee640f2d25c449ab8832f7533540eed"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "acb02d8bd9c1dabe45855fe221b25d76"
server_key_iv: "0a4de67e7670c2256431eecd050bcd82"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.215483 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "bee640f2d25c449ab8832f7533540eed"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "acb02d8bd9c1dabe45855fe221b25d76"
server_key_iv: "0a4de67e7670c2256431eecd050bcd82"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.223115 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260923 18:05:32.227680 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.229918 19149 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260923 18:05:32.230172 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "bee640f2d25c449ab8832f7533540eed"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "acb02d8bd9c1dabe45855fe221b25d76"
server_key_iv: "0a4de67e7670c2256431eecd050bcd82"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.230351 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:32.246318 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:32.251410 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:32.251695 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:32.261937 19149 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:33885
I20260923 18:05:32.262024 19216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:33885 every 8 connection(s)
I20260923 18:05:32.263409 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:32.267387 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:32.268772 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19149
I20260923 18:05:32.268904 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:32.269194 18611 external_mini_cluster.cc:1428] Setting key 869a07a1f3ebf0946faf75c80b98775c
I20260923 18:05:32.275154 19217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: Bootstrap starting.
I20260923 18:05:32.277813 19217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:32.278630 19217 log.cc:824] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:32.281993 19217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: No bootstrap required, opened a new log
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:32.285655 19217 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } }
I20260923 18:05:32.286010 19217 raft_consensus.cc:374] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:32.286087 19217 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee640f2d25c449ab8832f7533540eed, State: Initialized, Role: FOLLOWER
I20260923 18:05:32.286592 19217 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } }
I20260923 18:05:32.286759 19217 raft_consensus.cc:388] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:32.286852 19217 raft_consensus.cc:482] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:32.286959 19217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:32.288128 19217 raft_consensus.cc:504] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } }
I20260923 18:05:32.288527 19217 leader_election.cc:302] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bee640f2d25c449ab8832f7533540eed; no voters:
I20260923 18:05:32.288940 19217 leader_election.cc:288] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:32.289096 19222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:32.289479 19222 raft_consensus.cc:686] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [term 1 LEADER]: Becoming Leader. State: Replica: bee640f2d25c449ab8832f7533540eed, State: Running, Role: LEADER
I20260923 18:05:32.290192 19222 consensus_queue.cc:237] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } }
I20260923 18:05:32.290865 19217 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:32.291158 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:32.271368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.18.44.254:33885)
0923 18:05:32.272057 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:32.272079 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:32.272139 (+ 60us) server_negotiation.cc:407] Connection header received
0923 18:05:32.273134 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:32.273165 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:32.273695 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:32.274177 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:32.275425 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.276842 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.278059 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.278775 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.278977 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:32.283422 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:32.283431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:32.283463 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:32.283515 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:32.286477 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.287398 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.287409 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.287416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.287682 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.288056 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.288060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.288062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.288750 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:32.289059 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:32.289461 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 18:05:32.289908 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":179,"threads_started":1}
I20260923 18:05:32.291778 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bee640f2d25c449ab8832f7533540eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } } }
I20260923 18:05:32.291945 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:32.292200 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bee640f2d25c449ab8832f7533540eed. Latest consensus state: current_term: 1 leader_uuid: "bee640f2d25c449ab8832f7533540eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee640f2d25c449ab8832f7533540eed" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33885 } } }
I20260923 18:05:32.292323 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:32.297525 19237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:32.297703 19237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:32.297871 19229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:32.299175 19229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:32.309549 19229 catalog_manager.cc:1347] Generated new cluster ID: 8a21e6effc824863aca0e6f3279bd911
I20260923 18:05:32.309665 19229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:32.329862 19229 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:32.330976 19229 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:32.339807 19229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bee640f2d25c449ab8832f7533540eed: Generated new TSK 0
I20260923 18:05:32.341161 19229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:32.384836 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:32.512990 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:32.513325 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:32.513397 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:32.518007 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:32.518123 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:32.518227 19243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:32.524170 19243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:32.525599 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:32.526760 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:32.539434 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.539434 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.539434 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:32.539882 19243 server_base.cc:1034] running on GCE node
I20260923 18:05:32.540529 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:32.541320 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:32.542701 19243 hybrid_clock.cc:648] HybridClock initialized: now 1790186732542647 us; error 65 us; skew 500 ppm
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186732, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:32.546587 19243 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:32.547931 19243 webserver.cc:460] Webserver started at http://127.18.44.193:34139/ using document root <none> and password file <none>
I20260923 18:05:32.548619 19243 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:32.548707 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:32.548936 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:32.551321 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "686abdb7fa6645559133f228c6d50caa"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "9786c0b8c8ad59edd70cea267a5ab8f3"
server_key_iv: "7b68c5590dd60f039f75be1c84616bcf"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.551977 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "686abdb7fa6645559133f228c6d50caa"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "9786c0b8c8ad59edd70cea267a5ab8f3"
server_key_iv: "7b68c5590dd60f039f75be1c84616bcf"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.556943 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:32.560684 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.562670 19243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:32.562865 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "686abdb7fa6645559133f228c6d50caa"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "9786c0b8c8ad59edd70cea267a5ab8f3"
server_key_iv: "7b68c5590dd60f039f75be1c84616bcf"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.562999 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:32.585093 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:32.588490 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:32.588690 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:32.589327 19243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:32.590261 19243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:32.590314 19243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.590370 19243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:32.590399 19243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.602303 19243 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:35049
I20260923 18:05:32.602608 19371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:35049 every 8 connection(s)
I20260923 18:05:32.603708 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:32.605731 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19243
I20260923 18:05:32.605858 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:32.606142 18611 external_mini_cluster.cc:1428] Setting key bdacea92e28773c7fd26c00c507092d9
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186732, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:32.626811 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:32.607833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:55923 (local address 127.18.44.254:33885)
0923 18:05:32.608027 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:32.608033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:32.608053 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:05:32.609901 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:32.609905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:32.610043 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:32.610130 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:32.612025 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.612635 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.614031 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.614285 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.614385 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:32.621309 (+ 6924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:32.621314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:32.621316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:32.621356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:32.623482 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.624780 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.624785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.624787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.624899 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.625351 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.625355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.625356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.625603 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:32.625701 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:32.626411 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 18:05:32.626606 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:05:32.628224 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:32.606940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33885 (local address 127.18.44.193:55923)
0923 18:05:32.607657 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:32.607753 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:32.609436 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:32.610439 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:32.610451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:32.611018 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:32.611821 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:32.611843 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.612789 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.612792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:32.613883 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:32.613896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.614479 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.614483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:32.614697 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:32.615734 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:32.615757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:32.620967 (+ 5210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:32.623867 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:32.623876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:32.623900 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:32.624476 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:32.625002 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:32.625006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:32.625008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:32.625233 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:32.625718 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:32.625727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:32.626103 (+ 376us) client_negotiation.cc:769] Sending connection context
0923 18:05:32.626333 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 18:05:32.626851 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":142,"threads_started":1}
I20260923 18:05:32.630645 19372 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33885
I20260923 18:05:32.631060 19372 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:32.631932 19372 heartbeater.cc:511] Master 127.18.44.254:33885 requested a full tablet report, sending...
I20260923 18:05:32.634575 19180 ts_manager.cc:194] Registered new tserver with Master: 686abdb7fa6645559133f228c6d50caa (127.18.44.193:35049)
I20260923 18:05:32.636427 19180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:55923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:32.640466 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:32.760960 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:32.761296 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:32.761358 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:32.766027 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:32.766153 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:32.766247 19377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:32.772984 19377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:32.774113 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:32.775143 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:05:32.786543 19377 server_base.cc:1034] running on GCE node
W20260923 18:05:32.786487 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.786768 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:32.786787 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:32.787575 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:32.788442 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:32.789871 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790186732789858 us; error 54 us; skew 500 ppm
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186732, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:32.801103 19377 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:32.802347 19377 webserver.cc:460] Webserver started at http://127.18.44.194:44317/ using document root <none> and password file <none>
I20260923 18:05:32.802999 19377 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:32.803081 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:32.803352 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:32.805608 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "813f7ce772864cf9921e4dc51cd583f1"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "cfedd5e2f53ed51086b490f34ba4f821"
server_key_iv: "2909c88eaf648dce20c49ce5706a5d2e"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.806211 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "813f7ce772864cf9921e4dc51cd583f1"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "cfedd5e2f53ed51086b490f34ba4f821"
server_key_iv: "2909c88eaf648dce20c49ce5706a5d2e"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.810755 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:05:32.814103 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.816071 19377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:32.816246 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "813f7ce772864cf9921e4dc51cd583f1"
format_stamp: "Formatted at 2026-09-23 18:05:32 on dist-test-slave-rvjj"
server_key: "cfedd5e2f53ed51086b490f34ba4f821"
server_key_iv: "2909c88eaf648dce20c49ce5706a5d2e"
server_key_version: "encryptionkey@0"
I20260923 18:05:32.816380 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:32.831815 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:32.835778 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:32.836015 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:32.836683 19377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:32.837688 19377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:32.837745 19377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.837808 19377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:32.837841 19377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:32.849304 19377 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:40633
I20260923 18:05:32.849452 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:40633 every 8 connection(s)
I20260923 18:05:32.850579 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:32.860002 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19377
I20260923 18:05:32.860155 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:32.860469 18611 external_mini_cluster.cc:1428] Setting key e5c7ffc8df14ff3aac9ebad9618ed20b
Sep 23 18:05:32 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186732, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:32.869413 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:32.853077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:34197 (local address 127.18.44.254:33885)
0923 18:05:32.853356 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:32.853362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:32.854897 (+ 1535us) server_negotiation.cc:407] Connection header received
0923 18:05:32.856247 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:32.856251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:32.856329 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:32.856473 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:32.858438 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.859117 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.861446 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.861724 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.861846 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:32.865144 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:32.865148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:32.865150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:32.865190 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:32.867057 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.867677 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.867681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.867682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.867787 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:32.868175 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:32.868179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:32.868180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:32.868387 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:32.868484 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:32.869082 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 18:05:32.869235 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 18:05:32.870383 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:32.853544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33885 (local address 127.18.44.194:34197)
0923 18:05:32.854661 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:32.854733 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:32.855940 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:32.856862 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:32.856873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:32.857429 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:32.858158 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:32.858180 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.859391 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.859395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:32.861195 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:32.861212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:32.862077 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:32.862085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:32.862372 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:32.863059 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:32.863078 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:32.864856 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:32.867188 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:32.867196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:32.867217 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:32.867560 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:32.867865 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:32.867869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:32.867871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:32.868072 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:32.868486 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:32.868494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:32.868824 (+ 330us) client_negotiation.cc:769] Sending connection context
0923 18:05:32.869012 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 18:05:32.869405 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":167,"threads_started":1}
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:05:32.871878 19506 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33885
I20260923 18:05:32.872277 19506 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:32.872987 19506 heartbeater.cc:511] Master 127.18.44.254:33885 requested a full tablet report, sending...
I20260923 18:05:32.874583 19180 ts_manager.cc:194] Registered new tserver with Master: 813f7ce772864cf9921e4dc51cd583f1 (127.18.44.194:40633)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:32.875622 19180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:34197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:32.897253 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:33.022749 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:33.023105 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:33.023173 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:33.028242 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:33.028368 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:33.028468 19511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:33.035099 19511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:33.036379 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:33.037607 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:33.049712 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:33.049923 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:33.050062 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:33.050251 19511 server_base.cc:1034] running on GCE node
I20260923 18:05:33.050841 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:33.051891 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:33.053130 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790186733053088 us; error 61 us; skew 500 ppm
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:33.057406 19511 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:33.058785 19511 webserver.cc:460] Webserver started at http://127.18.44.195:45239/ using document root <none> and password file <none>
I20260923 18:05:33.059648 19511 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:33.059770 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:33.060056 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:33.062810 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "b04ea68651724baeb21029a012976bc2"
format_stamp: "Formatted at 2026-09-23 18:05:33 on dist-test-slave-rvjj"
server_key: "d1c3cb5039580459f1ea975a1df679b8"
server_key_iv: "b177d17c5cf7a887dbc6fab3414c6de8"
server_key_version: "encryptionkey@0"
I20260923 18:05:33.063611 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "b04ea68651724baeb21029a012976bc2"
format_stamp: "Formatted at 2026-09-23 18:05:33 on dist-test-slave-rvjj"
server_key: "d1c3cb5039580459f1ea975a1df679b8"
server_key_iv: "b177d17c5cf7a887dbc6fab3414c6de8"
server_key_version: "encryptionkey@0"
I20260923 18:05:33.069276 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 18:05:33.073153 19526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:33.075121 19511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:33.075390 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "b04ea68651724baeb21029a012976bc2"
format_stamp: "Formatted at 2026-09-23 18:05:33 on dist-test-slave-rvjj"
server_key: "d1c3cb5039580459f1ea975a1df679b8"
server_key_iv: "b177d17c5cf7a887dbc6fab3414c6de8"
server_key_version: "encryptionkey@0"
I20260923 18:05:33.075570 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:33.097388 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:33.103147 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:33.103542 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:33.104501 19511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:33.105850 19511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:33.105921 19511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:33.105978 19511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:33.106005 19511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:33.119395 19511 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:37731
I20260923 18:05:33.119439 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:37731 every 8 connection(s)
I20260923 18:05:33.120690 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:33.126652 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19511
I20260923 18:05:33.126804 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:33.127074 18611 external_mini_cluster.cc:1428] Setting key fbe9e17a13722e73dbc0bd7037dc5392
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:33.141780 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.123666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:38621 (local address 127.18.44.254:33885)
0923 18:05:33.123965 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.123970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.125479 (+ 1509us) server_negotiation.cc:407] Connection header received
0923 18:05:33.126722 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.126728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.126831 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.126970 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:33.130210 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.130966 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.132690 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.132956 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.133067 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.136834 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:33.136839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:33.136841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:33.136881 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:33.138428 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.139436 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.139439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.139441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.139547 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.140103 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.140107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.140108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.140391 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:33.140489 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.141385 (+ 896us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.141612 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260923 18:05:33.143241 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.124205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33885 (local address 127.18.44.195:38621)
0923 18:05:33.125259 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.125323 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.126343 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.127371 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.127384 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.128380 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:33.129861 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.129888 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.131156 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.131160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.132470 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.132490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.133210 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.133214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.133446 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.134229 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:33.134251 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:33.136478 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:33.138690 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:33.138700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:33.138725 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:33.139196 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:33.139702 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:33.139705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:33.139707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:33.139948 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:33.140559 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:33.140568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:33.141039 (+ 471us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.141284 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.141776 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":189,"threads_started":1}
I20260923 18:05:33.145116 19640 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33885
I20260923 18:05:33.145495 19640 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:33.146341 19640 heartbeater.cc:511] Master 127.18.44.254:33885 requested a full tablet report, sending...
I20260923 18:05:33.148062 19180 ts_manager.cc:194] Registered new tserver with Master: b04ea68651724baeb21029a012976bc2 (127.18.44.195:37731)
I20260923 18:05:33.148876 19180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:38621
I20260923 18:05:33.156791 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:05:33.171914 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.162059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.18.44.254:33885)
0923 18:05:33.162266 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.162270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.162366 (+ 96us) server_negotiation.cc:407] Connection header received
0923 18:05:33.162576 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.162580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.162637 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.162813 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:33.163849 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.164416 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.165538 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.165779 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.165895 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.168058 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:33.168061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:33.168064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:33.168096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:33.169565 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.170284 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.170287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.170288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.170385 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.170812 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.170816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.170818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.171075 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:33.171325 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.171639 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.171765 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 18:05:33.180478 19180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:33.185942 19180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:33.206733 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.199618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.18.44.195:37731)
0923 18:05:33.200123 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.200136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.200792 (+ 656us) server_negotiation.cc:407] Connection header received
0923 18:05:33.201322 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.201330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.201606 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.201808 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:33.202255 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.203433 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.204992 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.205801 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.205941 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.206171 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.206366 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.206501 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":151,"threads_started":1}
I20260923 18:05:33.206728 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.199353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35049 (local address 127.0.0.1:57366)
0923 18:05:33.200399 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.200462 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.200617 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.201578 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.201585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.201610 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:33.201728 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.201734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.202941 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.202944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.204403 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.204417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.205824 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.205829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.206002 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.206051 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.206202 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.206439 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":1,"thread_start_us":173,"threads_started":1}
I20260923 18:05:33.206782 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.199798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.18.44.193:35049)
0923 18:05:33.200292 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.200301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.200813 (+ 512us) server_negotiation.cc:407] Connection header received
0923 18:05:33.200929 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.200935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.201204 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.201368 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:33.201877 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.202796 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.204655 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.205581 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.205717 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.205930 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.206449 (+ 519us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.206587 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":143,"threads_started":1}
I20260923 18:05:33.207125 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.200534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:40633 (local address 127.0.0.1:35092)
0923 18:05:33.201027 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.201040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.201127 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.201791 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.201793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.201810 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:33.201896 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.201902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.203521 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.203524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.204850 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.204863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.206371 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.206374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.206463 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.206486 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.206552 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.206632 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":111,"threads_started":1}
I20260923 18:05:33.207696 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.200750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.18.44.194:40633)
0923 18:05:33.201284 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.201299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.201326 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:05:33.201402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.201406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.201606 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.201808 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:33.201999 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.203326 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.205055 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.206026 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.206145 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.206309 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.206814 (+ 505us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.207571 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":152,"threads_started":1}
I20260923 18:05:33.208087 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.199353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:37731 (local address 127.0.0.1:50256)
0923 18:05:33.200459 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.200477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.200639 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.202009 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.202012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.202026 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:33.202113 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.202118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.203614 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.203619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.204866 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.204878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.206058 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.206062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.206170 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.206216 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.206308 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.207966 (+ 1658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":1623,"thread_start_us":135,"threads_started":1}
I20260923 18:05:33.211169 19440 tablet_service.cc:1467] Processing CreateTablet for tablet 247b70bd42c24c268fbc1ec07bbbbce6 (DEFAULT_TABLE table=test-table [id=0807286e80c540fca2a4f258d2984020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:33.211601 19574 tablet_service.cc:1467] Processing CreateTablet for tablet 247b70bd42c24c268fbc1ec07bbbbce6 (DEFAULT_TABLE table=test-table [id=0807286e80c540fca2a4f258d2984020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:33.211290 19306 tablet_service.cc:1467] Processing CreateTablet for tablet 247b70bd42c24c268fbc1ec07bbbbce6 (DEFAULT_TABLE table=test-table [id=0807286e80c540fca2a4f258d2984020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:33.212863 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247b70bd42c24c268fbc1ec07bbbbce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:33.212884 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247b70bd42c24c268fbc1ec07bbbbce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:33.212884 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247b70bd42c24c268fbc1ec07bbbbce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:33.222148 19657 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Bootstrap starting.
I20260923 18:05:33.223603 19658 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Bootstrap starting.
I20260923 18:05:33.225945 19659 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Bootstrap starting.
I20260923 18:05:33.226240 19657 tablet_bootstrap.cc:654] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:33.226691 19658 tablet_bootstrap.cc:654] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:33.227516 19658 log.cc:824] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:33.227517 19657 log.cc:824] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:33.229492 19659 tablet_bootstrap.cc:654] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:33.230149 19658 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: No bootstrap required, opened a new log
I20260923 18:05:33.230158 19657 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: No bootstrap required, opened a new log
I20260923 18:05:33.230407 19657 ts_tablet_manager.cc:1397] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 18:05:33.230407 19658 ts_tablet_manager.cc:1397] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:05:33.231457 19659 log.cc:824] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:33.234565 19659 tablet_bootstrap.cc:492] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: No bootstrap required, opened a new log
I20260923 18:05:33.234881 19659 ts_tablet_manager.cc:1397] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260923 18:05:33.235514 19658 raft_consensus.cc:348] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.235749 19657 raft_consensus.cc:348] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.235970 19657 raft_consensus.cc:374] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:33.235970 19658 raft_consensus.cc:374] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:33.236037 19657 raft_consensus.cc:729] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686abdb7fa6645559133f228c6d50caa, State: Initialized, Role: FOLLOWER
I20260923 18:05:33.236037 19658 raft_consensus.cc:729] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813f7ce772864cf9921e4dc51cd583f1, State: Initialized, Role: FOLLOWER
I20260923 18:05:33.236603 19657 consensus_queue.cc:260] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.236603 19658 consensus_queue.cc:260] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.237892 19657 ts_tablet_manager.cc:1428] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260923 18:05:33.237892 19658 ts_tablet_manager.cc:1428] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 18:05:33.238122 19372 heartbeater.cc:503] Master 127.18.44.254:33885 was elected leader, sending a full tablet report...
I20260923 18:05:33.238423 19506 heartbeater.cc:503] Master 127.18.44.254:33885 was elected leader, sending a full tablet report...
I20260923 18:05:33.240327 19659 raft_consensus.cc:348] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.240765 19659 raft_consensus.cc:374] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:33.240844 19659 raft_consensus.cc:729] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04ea68651724baeb21029a012976bc2, State: Initialized, Role: FOLLOWER
I20260923 18:05:33.241411 19659 consensus_queue.cc:260] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.242815 19640 heartbeater.cc:503] Master 127.18.44.254:33885 was elected leader, sending a full tablet report...
I20260923 18:05:33.242865 19659 ts_tablet_manager.cc:1428] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
W20260923 18:05:33.352159 19507 tablet.cc:2369] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:05:33.355490 19373 tablet.cc:2369] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:05:33.362185 19664 raft_consensus.cc:482] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:33.362432 19664 raft_consensus.cc:504] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.364110 19664 leader_election.cc:288] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b04ea68651724baeb21029a012976bc2 (127.18.44.195:37731), 686abdb7fa6645559133f228c6d50caa (127.18.44.193:35049)
I20260923 18:05:33.369292 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.364689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:38219 (local address 127.18.44.195:37731)
0923 18:05:33.364860 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.364866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.364889 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:05:33.365037 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.365040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.365114 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.365228 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:33.365592 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.366126 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.367799 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.368590 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.368834 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.368898 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.369059 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.369132 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 18:05:33.369335 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.364385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:37731 (local address 127.18.44.194:38219)
0923 18:05:33.364601 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.364622 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.364759 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.365282 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.365295 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.365324 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:33.365461 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.365468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.366384 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.366392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.367612 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.367629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.368791 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.368796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.368918 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.368944 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.369026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.369130 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"spinlock_wait_cycles":7936}
I20260923 18:05:33.369696 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.364688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:44655 (local address 127.18.44.193:35049)
0923 18:05:33.364860 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.364866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.365343 (+ 477us) server_negotiation.cc:407] Connection header received
0923 18:05:33.365485 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.365487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.365543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.365623 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:33.365934 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.366804 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.368108 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.368928 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.369110 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.369195 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.369387 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.369517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:05:33.370326 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.364581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35049 (local address 127.18.44.194:44655)
0923 18:05:33.365163 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:33.365195 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:33.365361 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:33.365644 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:33.365647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:33.365663 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:33.365817 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.365824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.366943 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.366947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.367951 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:33.367965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.369113 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.369117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:33.369238 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.369266 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:05:33.369349 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 18:05:33.370061 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"spinlock_wait_cycles":896,"thread_start_us":150,"threads_started":1}
I20260923 18:05:33.370397 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247b70bd42c24c268fbc1ec07bbbbce6" candidate_uuid: "813f7ce772864cf9921e4dc51cd583f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04ea68651724baeb21029a012976bc2" is_pre_election: true
I20260923 18:05:33.370816 19594 raft_consensus.cc:2454] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813f7ce772864cf9921e4dc51cd583f1 in term 0.
I20260923 18:05:33.371517 19308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247b70bd42c24c268fbc1ec07bbbbce6" candidate_uuid: "813f7ce772864cf9921e4dc51cd583f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686abdb7fa6645559133f228c6d50caa" is_pre_election: true
I20260923 18:05:33.371443 19394 leader_election.cc:302] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813f7ce772864cf9921e4dc51cd583f1, b04ea68651724baeb21029a012976bc2; no voters:
W20260923 18:05:33.371937 19641 tablet.cc:2369] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:05:33.371968 19308 raft_consensus.cc:2454] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813f7ce772864cf9921e4dc51cd583f1 in term 0.
I20260923 18:05:33.371982 19664 raft_consensus.cc:2781] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:33.372269 19664 raft_consensus.cc:482] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:33.372331 19664 raft_consensus.cc:3037] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:33.373790 19664 raft_consensus.cc:504] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.374261 19664 leader_election.cc:288] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [CANDIDATE]: Term 1 election: Requested vote from peers b04ea68651724baeb21029a012976bc2 (127.18.44.195:37731), 686abdb7fa6645559133f228c6d50caa (127.18.44.193:35049)
I20260923 18:05:33.375051 19308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247b70bd42c24c268fbc1ec07bbbbce6" candidate_uuid: "813f7ce772864cf9921e4dc51cd583f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686abdb7fa6645559133f228c6d50caa"
I20260923 18:05:33.375092 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247b70bd42c24c268fbc1ec07bbbbce6" candidate_uuid: "813f7ce772864cf9921e4dc51cd583f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04ea68651724baeb21029a012976bc2"
I20260923 18:05:33.375379 19594 raft_consensus.cc:3037] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:33.375372 19308 raft_consensus.cc:3037] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:33.376945 19308 raft_consensus.cc:2454] T 247b70bd42c24c268fbc1ec07bbbbce6 P 686abdb7fa6645559133f228c6d50caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813f7ce772864cf9921e4dc51cd583f1 in term 1.
I20260923 18:05:33.377389 19594 raft_consensus.cc:2454] T 247b70bd42c24c268fbc1ec07bbbbce6 P b04ea68651724baeb21029a012976bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813f7ce772864cf9921e4dc51cd583f1 in term 1.
I20260923 18:05:33.377547 19395 leader_election.cc:302] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686abdb7fa6645559133f228c6d50caa, 813f7ce772864cf9921e4dc51cd583f1; no voters:
I20260923 18:05:33.377837 19664 raft_consensus.cc:2781] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:33.378129 19664 raft_consensus.cc:686] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [term 1 LEADER]: Becoming Leader. State: Replica: 813f7ce772864cf9921e4dc51cd583f1, State: Running, Role: LEADER
I20260923 18:05:33.378742 19664 consensus_queue.cc:237] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } }
I20260923 18:05:33.383298 19180 catalog_manager.cc:5495] T 247b70bd42c24c268fbc1ec07bbbbce6 P 813f7ce772864cf9921e4dc51cd583f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 813f7ce772864cf9921e4dc51cd583f1 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "813f7ce772864cf9921e4dc51cd583f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ea68651724baeb21029a012976bc2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 37731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686abdb7fa6645559133f228c6d50caa" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813f7ce772864cf9921e4dc51cd583f1" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 40633 } health_report { overall_health: HEALTHY } } }
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:33.414791 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.401729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.18.44.193:35049)
0923 18:05:33.401994 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.401998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.402016 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:05:33.402069 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.402072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.402118 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.402213 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:33.403370 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.404042 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.405465 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.405854 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.405989 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.410023 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:33.410041 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:33.410057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:33.410101 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:33.412015 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.412726 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.412733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.412734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.412841 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.413397 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.413401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.413402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.413717 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:33.413959 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.414469 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.414617 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:33.453665 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.439806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.18.44.193:35049)
0923 18:05:33.439974 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.439978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.440000 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:05:33.440265 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.440268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.440326 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.440446 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:33.441882 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.442484 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.444112 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.444391 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.444516 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.448735 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:33.448740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:33.448742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:33.448782 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:33.450378 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.451154 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.451157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.451158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.451330 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.451871 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.451875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.451876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.452143 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:33.452271 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.453297 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.453485 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:33.488485 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:33.476782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.18.44.193:35049)
0923 18:05:33.477057 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:33.477063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:33.477285 (+ 222us) server_negotiation.cc:407] Connection header received
0923 18:05:33.477468 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:33.477472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:33.477533 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:33.477624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:33.478505 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.479082 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.480520 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:33.480809 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:33.481110 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:33.484463 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:33.484469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:33.484472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:33.484520 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:33.486209 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.486943 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.486946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.486948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.487071 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:33.487422 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:33.487426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:33.487427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:33.487683 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:33.487826 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:33.488109 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 18:05:33.488287 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260923 18:05:33.491230 19286 tablet_service.cc:2908] Rejecting scan request for tablet 247b70bd42c24c268fbc1ec07bbbbce6: Uninitialized: clean time has not yet been initialized
I20260923 18:05:33.495237 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19243
I20260923 18:05:33.501421 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19377
I20260923 18:05:33.509683 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19511
I20260923 18:05:33.517285 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19149
2026-09-23T18:05:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2653 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19681](info): set up 2 sockets
Sep 23 18:05:33 dist-test-slave-rvjj krb5kdc[19681](info): commencing operation
krb5kdc: starting...
W20260923 18:05:34.540850 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:34 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:34.731911 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33047
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:34.872227 19694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:34.872581 19694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:34.872639 19694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:34.877293 19694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:34.877418 19694 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:05:34.877447 19694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:34.877485 19694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:34.877506 19694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:34.877530 19694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:34.883929 19694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:33047
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:34.885177 19694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:34.886303 19694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:34.896783 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:34.896783 19702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:34.896951 19700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:34.896778 19694 server_base.cc:1034] running on GCE node
I20260923 18:05:34.897833 19694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:34.899123 19694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:34.902513 19694 hybrid_clock.cc:648] HybridClock initialized: now 1790186734902464 us; error 62 us; skew 500 ppm
Sep 23 18:05:34 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186734, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:34.906117 19694 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:34.907648 19694 webserver.cc:460] Webserver started at http://127.18.44.254:40409/ using document root <none> and password file <none>
I20260923 18:05:34.908345 19694 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:34.908425 19694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:34.908639 19694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:34.910987 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "f4b8ac97d83b402798c8daa0d6b94734"
format_stamp: "Formatted at 2026-09-23 18:05:34 on dist-test-slave-rvjj"
server_key: "ca9bad42b567b7c94844a33f3bf74357"
server_key_iv: "c7af4a6082b8677bdd775936003e2357"
server_key_version: "encryptionkey@0"
I20260923 18:05:34.911670 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "f4b8ac97d83b402798c8daa0d6b94734"
format_stamp: "Formatted at 2026-09-23 18:05:34 on dist-test-slave-rvjj"
server_key: "ca9bad42b567b7c94844a33f3bf74357"
server_key_iv: "c7af4a6082b8677bdd775936003e2357"
server_key_version: "encryptionkey@0"
I20260923 18:05:34.916663 19694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:34.920087 19709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:34.921785 19694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:34.922034 19694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "f4b8ac97d83b402798c8daa0d6b94734"
format_stamp: "Formatted at 2026-09-23 18:05:34 on dist-test-slave-rvjj"
server_key: "ca9bad42b567b7c94844a33f3bf74357"
server_key_iv: "c7af4a6082b8677bdd775936003e2357"
server_key_version: "encryptionkey@0"
I20260923 18:05:34.922180 19694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:34.938869 19694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:34.943887 19694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:34.944123 19694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:34.953673 19694 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:33047
I20260923 18:05:34.953743 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:33047 every 8 connection(s)
I20260923 18:05:34.954815 19694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:34.958982 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:34.962993 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19694
I20260923 18:05:34.963168 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:34.963564 18611 external_mini_cluster.cc:1428] Setting key e0b187689f4d9de3626e891511dd697d
I20260923 18:05:34.969409 19772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Bootstrap starting.
I20260923 18:05:34.973347 19772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:34.974511 19772 log.cc:824] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:05:34 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:34.977626 19772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: No bootstrap required, opened a new log
I20260923 18:05:34.981262 19772 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } }
I20260923 18:05:34.981598 19772 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:34.981635 19772 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b8ac97d83b402798c8daa0d6b94734, State: Initialized, Role: FOLLOWER
I20260923 18:05:34.982196 19772 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } }
I20260923 18:05:34.982363 19772 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:34.982442 19772 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:34.982550 19772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:34.983850 19772 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } }
I20260923 18:05:34.984292 19772 leader_election.cc:302] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b8ac97d83b402798c8daa0d6b94734; no voters:
I20260923 18:05:34.984833 19772 leader_election.cc:288] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:34.984990 19777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:34.985296 19777 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [term 1 LEADER]: Becoming Leader. State: Replica: f4b8ac97d83b402798c8daa0d6b94734, State: Running, Role: LEADER
I20260923 18:05:34.986068 19777 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } }
I20260923 18:05:34.986195 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:34.965528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.18.44.254:33047)
0923 18:05:34.966105 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:34.966122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:34.966162 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:05:34.967497 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:34.967525 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:34.967978 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:34.968350 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:34.969827 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:34.971069 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:34.972654 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:34.973439 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:34.973672 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:34.976772 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:34.976782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:34.976805 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:34.976853 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:34.980375 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:34.981118 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:34.981127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:34.981135 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:34.981342 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:34.981848 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:34.981852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:34.981853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:34.982688 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:34.983036 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:34.983559 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 18:05:34.984182 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":222,"threads_started":1}
I20260923 18:05:34.986717 19772 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:34.988444 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4b8ac97d83b402798c8daa0d6b94734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } } }
I20260923 18:05:34.988759 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:34.990248 19791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:34.989931 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b8ac97d83b402798c8daa0d6b94734. Latest consensus state: current_term: 1 leader_uuid: "f4b8ac97d83b402798c8daa0d6b94734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b8ac97d83b402798c8daa0d6b94734" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33047 } } }
I20260923 18:05:34.990589 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:34.995806 19791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:35.004864 19791 catalog_manager.cc:1347] Generated new cluster ID: 296634f6502c4a6fae5386727b091cde
I20260923 18:05:35.005000 19791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:35.018466 19791 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:35.019726 19791 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:35.029944 19791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Generated new TSK 0
I20260923 18:05:35.031440 19791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:35.079878 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33047
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:35.221498 19798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:35.221848 19798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:35.221904 19798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:35.227032 19798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:35.227124 19798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:35.227169 19798 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:35.227203 19798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:35.233212 19798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:35.234422 19798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:35.235581 19798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:05:35.248548 19798 server_base.cc:1034] running on GCE node
W20260923 18:05:35.248576 19804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:35.248576 19803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:35.248879 19806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:35.249431 19798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:35.250186 19798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:35.251379 19798 hybrid_clock.cc:648] HybridClock initialized: now 1790186735251360 us; error 48 us; skew 500 ppm
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:35.255175 19798 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:35.256693 19798 webserver.cc:460] Webserver started at http://127.18.44.193:42777/ using document root <none> and password file <none>
I20260923 18:05:35.257629 19798 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:35.257828 19798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:35.258085 19798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:35.261215 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "181cd240b5e543a2ae5dea36712c7909"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "d18d04b0491a41d8180b5c068cdd6373"
server_key_iv: "d8c2cd2826c54451a071218ba9fcaf9a"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.262519 19798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "181cd240b5e543a2ae5dea36712c7909"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "d18d04b0491a41d8180b5c068cdd6373"
server_key_iv: "d8c2cd2826c54451a071218ba9fcaf9a"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.268527 19798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:05:35.272598 19813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.274657 19798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260923 18:05:35.274852 19798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "181cd240b5e543a2ae5dea36712c7909"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "d18d04b0491a41d8180b5c068cdd6373"
server_key_iv: "d8c2cd2826c54451a071218ba9fcaf9a"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.275007 19798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:35.289788 19798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:35.295401 19798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:35.295995 19798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:35.298638 19798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:35.298753 19798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.298827 19798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:35.298857 19798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:35.321527 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:38751 every 8 connection(s)
I20260923 18:05:35.321480 19798 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:38751
I20260923 18:05:35.326578 19798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:35.328387 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.299572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38183 (local address 127.18.44.254:33047)
0923 18:05:35.299788 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.299794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.301863 (+ 2069us) server_negotiation.cc:407] Connection header received
0923 18:05:35.303852 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.303859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.303971 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.304191 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.306354 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.308966 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.310966 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.311611 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.311820 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.317578 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.317588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.317591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.317889 (+ 298us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.321095 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.322150 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.322154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.322156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.322267 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.323422 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.323426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.323428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.323713 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.323835 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.327649 (+ 3814us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.328195 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:05:35.329509 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19798
I20260923 18:05:35.329620 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:35.329646 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.300700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.193:38183)
0923 18:05:35.301458 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.301606 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.303341 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.304575 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.304591 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.305177 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.305849 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.305869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.309259 (+ 3390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.309263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.310720 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.310743 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.312039 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.312043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.312373 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.313863 (+ 1490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.313890 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.317072 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.321477 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.321486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.321518 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.321949 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.322724 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.322730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.322733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.323065 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.325489 (+ 2424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.325501 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.327017 (+ 1516us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.327432 (+ 415us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.328167 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":195,"threads_started":1}
I20260923 18:05:35.329923 18611 external_mini_cluster.cc:1428] Setting key fba72e9a63306bf23221762ca6f74959
I20260923 18:05:35.331749 19929 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33047
I20260923 18:05:35.332100 19929 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:35.332818 19929 heartbeater.cc:511] Master 127.18.44.254:33047 requested a full tablet report, sending...
I20260923 18:05:35.336081 19723 ts_manager.cc:194] Registered new tserver with Master: 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751)
I20260923 18:05:35.338088 19723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:38183
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:35.351886 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.339184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.193:35707)
0923 18:05:35.339965 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.339987 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.340119 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.340474 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.340477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.340769 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.340910 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.340921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.342806 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.342811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.343857 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.343876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.344885 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.344889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.345008 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.345799 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.345817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.346317 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.349259 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.349281 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.349283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.349610 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.350262 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.350266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.350269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.350501 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.351083 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.351087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.351216 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.351379 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.351648 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":158,"threads_started":1}
I20260923 18:05:35.351866 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.340090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35707 (local address 127.18.44.254:33047)
0923 18:05:35.340213 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.340217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.340232 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:05:35.340289 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.340292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.340340 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.340437 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.341675 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.342353 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.344242 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.344643 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.344819 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.346679 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.346683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.346686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.346818 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.348956 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.349932 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.349936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.349937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.350087 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.350642 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.350645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.350647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.350938 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.351072 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.351450 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.351647 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:35.382745 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33047
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:35.524142 19938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:35.524507 19938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:35.524571 19938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:35.530644 19938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:35.530759 19938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:35.530822 19938 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:35.530882 19938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:35.538275 19938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:35.539638 19938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:35.540751 19938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:35.554168 19946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:35.554251 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:35.554251 19938 server_base.cc:1034] running on GCE node
W20260923 18:05:35.554490 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:35.555217 19938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:35.556079 19938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:35.559628 19938 hybrid_clock.cc:648] HybridClock initialized: now 1790186735559527 us; error 116 us; skew 500 ppm
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:35.563337 19938 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:35.564489 19938 webserver.cc:460] Webserver started at http://127.18.44.194:37923/ using document root <none> and password file <none>
I20260923 18:05:35.565299 19938 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:35.565400 19938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:35.565635 19938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:35.568205 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "e6b7d6b6af504841a240cd7ab53112e9"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "78ce1ba8e422ec306ea38e43b0cc31aa"
server_key_iv: "be7973320c3d4fe0d2140f812f860851"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.568861 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "e6b7d6b6af504841a240cd7ab53112e9"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "78ce1ba8e422ec306ea38e43b0cc31aa"
server_key_iv: "be7973320c3d4fe0d2140f812f860851"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.574716 19938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 18:05:35.578509 19953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.580864 19938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:35.581084 19938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "e6b7d6b6af504841a240cd7ab53112e9"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "78ce1ba8e422ec306ea38e43b0cc31aa"
server_key_iv: "be7973320c3d4fe0d2140f812f860851"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.581225 19938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:35.597935 19938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:35.602406 19938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:35.602643 19938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:35.604557 19938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:35.604628 19938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.604686 19938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:35.604717 19938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.617074 19938 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:36497
I20260923 18:05:35.617127 20068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:36497 every 8 connection(s)
I20260923 18:05:35.618415 19938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:35.623631 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 19938
I20260923 18:05:35.623792 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:35.624118 18611 external_mini_cluster.cc:1428] Setting key 52e43182ce08c61a4489a4699ae61b80
I20260923 18:05:35.628386 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.606528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:45151 (local address 127.18.44.254:33047)
0923 18:05:35.606732 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.606736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.608156 (+ 1420us) server_negotiation.cc:407] Connection header received
0923 18:05:35.609693 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.609697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.609787 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.609881 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.612980 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.613841 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.616207 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.616561 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.616753 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.621038 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.621042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.621044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.621086 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.623806 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.625088 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.625093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.625095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.625255 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.626319 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.626323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.626325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.626574 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.626716 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.627853 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.628171 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260923 18:05:35.629777 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.606879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.194:45151)
0923 18:05:35.607678 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.607878 (+ 200us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.609367 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.610249 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.610551 (+ 302us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.611323 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.612589 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.612624 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.614179 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.614183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.615934 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.615970 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.616870 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.616876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.617253 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.618154 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.618177 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.620637 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.624212 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.624228 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.624263 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.624854 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.625412 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.625416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.625418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.625943 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.626860 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.626870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.627365 (+ 495us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.627618 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.628250 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":159,"threads_started":1}
I20260923 18:05:35.631912 20069 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33047
I20260923 18:05:35.632148 20069 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:35.632817 20069 heartbeater.cc:511] Master 127.18.44.254:33047 requested a full tablet report, sending...
I20260923 18:05:35.634449 19722 ts_manager.cc:194] Registered new tserver with Master: e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497)
I20260923 18:05:35.635491 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:45151
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:35.648789 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.636430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.194:47425)
0923 18:05:35.637065 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.637094 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.637311 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.637612 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.637616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.637906 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.638045 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.638054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.639016 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.639022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.639716 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.639730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.640240 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.640244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.640361 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.641153 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.641173 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.641891 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.646204 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.646210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.646212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.646531 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.647149 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.647153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.647155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.647323 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.648170 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.648173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.648305 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.648422 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.648589 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":150,"threads_started":1}
I20260923 18:05:35.649164 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.636556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:47425 (local address 127.18.44.254:33047)
0923 18:05:35.636718 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.636724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.637209 (+ 485us) server_negotiation.cc:407] Connection header received
0923 18:05:35.637430 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.637433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.637494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.637569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.638183 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.638838 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.639857 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.640104 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.640207 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.642684 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.642693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.642697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.642886 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.645517 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.646739 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.646744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.646746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.646953 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.647513 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.647518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.647520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.648007 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.648168 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.648634 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.648829 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:35.669062 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33047
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:35.791594 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:35.791913 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:35.791958 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:35.796504 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:35.796595 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:35.796639 20078 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:35.796677 20078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:35.802701 20078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:35.803952 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:35.805037 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:35.817965 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:35.818048 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:35.817953 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:35.818411 20078 server_base.cc:1034] running on GCE node
I20260923 18:05:35.819222 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:35.820299 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:35.821563 20078 hybrid_clock.cc:648] HybridClock initialized: now 1790186735821516 us; error 79 us; skew 500 ppm
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:35.826719 20078 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:35.828660 20078 webserver.cc:460] Webserver started at http://127.18.44.195:34661/ using document root <none> and password file <none>
I20260923 18:05:35.829766 20078 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:35.829898 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:35.830216 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:35.834132 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "20dabf8028f4471da81e0651f3cbe229"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "0e4f53210b65f35a41be0092f8c9e18d"
server_key_iv: "a7ba3c3bf78e19adb65c479e210cc3db"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.835306 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "20dabf8028f4471da81e0651f3cbe229"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "0e4f53210b65f35a41be0092f8c9e18d"
server_key_iv: "a7ba3c3bf78e19adb65c479e210cc3db"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.842154 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 18:05:35.847194 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.849370 20078 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 18:05:35.849639 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "20dabf8028f4471da81e0651f3cbe229"
format_stamp: "Formatted at 2026-09-23 18:05:35 on dist-test-slave-rvjj"
server_key: "0e4f53210b65f35a41be0092f8c9e18d"
server_key_iv: "a7ba3c3bf78e19adb65c479e210cc3db"
server_key_version: "encryptionkey@0"
I20260923 18:05:35.849846 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:35.877163 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:35.882072 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:35.882293 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:35.884483 20078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:35.884567 20078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:35.884624 20078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:35.884639 20078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:35.899473 20078 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:34459
I20260923 18:05:35.899749 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:34459 every 8 connection(s)
I20260923 18:05:35.901161 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:35.903861 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.885558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:44209 (local address 127.18.44.254:33047)
0923 18:05:35.885774 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.885782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.886923 (+ 1141us) server_negotiation.cc:407] Connection header received
0923 18:05:35.888464 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.888468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.888542 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.888624 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.890771 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.891771 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.893725 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.894125 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.894334 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.898218 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.898223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.898227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.898267 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.900384 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.901269 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.901275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.901277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.901469 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.901956 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.901960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.901962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.902233 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.902330 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.903189 (+ 859us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.903685 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:05:35.904489 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.885996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.195:44209)
0923 18:05:35.886631 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.886700 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.888178 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.888998 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.889009 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.889711 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.890472 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.890503 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.892022 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.892026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.893403 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.893423 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.894418 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.894423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.894804 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.895681 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.895702 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.897789 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.900609 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.900616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.900635 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.901042 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.901609 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.901611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.901613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.901818 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.902417 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.902422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.902762 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.902973 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.903426 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":169,"threads_started":1}
I20260923 18:05:35.906533 20209 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33047
I20260923 18:05:35.906944 20209 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:35.907928 20209 heartbeater.cc:511] Master 127.18.44.254:33047 requested a full tablet report, sending...
I20260923 18:05:35.909380 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 20078
I20260923 18:05:35.909425 19723 ts_manager.cc:194] Registered new tserver with Master: 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195:34459)
I20260923 18:05:35.909503 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:35.910102 18611 external_mini_cluster.cc:1428] Setting key 2465790b214fd9706b942ab8d2e3cba7
I20260923 18:05:35.910115 19723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:44209
I20260923 18:05:35.912773 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:05:35.921147 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.909037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:56617 (local address 127.18.44.254:33047)
0923 18:05:35.910775 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.910780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.911662 (+ 882us) server_negotiation.cc:407] Connection header received
0923 18:05:35.911903 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.911906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.911956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.912041 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.912740 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.913499 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.914729 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.915061 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.915182 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.916877 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.916881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.916883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.916913 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.918690 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.919376 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.919381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.919383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.919521 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.919963 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.919967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.919968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.920226 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.920335 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.920861 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.921002 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622}
I20260923 18:05:35.921392 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.908877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.18.44.195:56617)
0923 18:05:35.911553 (+ 2676us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.911571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.911791 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.912073 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.912076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.912328 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:35.912452 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.912460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.913718 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.913723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.914479 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.914499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.915219 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.915223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.915380 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.916186 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:35.916200 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:35.916660 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:35.918854 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.918858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.918861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.919169 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.919642 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:35.919645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:35.919648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:35.919755 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:35.920342 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:35.920345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:35.920494 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.920712 (+ 218us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.921042 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2572,"thread_start_us":178,"threads_started":1}
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:05:35 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:35.945775 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.934648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.18.44.254:33047)
0923 18:05:35.934833 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.934838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.934856 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:05:35.934916 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.934919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.934974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.935099 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:35.936149 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.936859 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.937925 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.938219 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.938361 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.941205 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:35.941211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:35.941214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:35.941266 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:35.943191 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.944040 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.944044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.944045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.944176 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:35.944640 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:35.944644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:35.944645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:35.944911 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:35.945055 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.945402 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.945561 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:05:35.950268 19723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:35.954118 19723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:35.972802 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.965729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:38751 (local address 127.0.0.1:40590)
0923 18:05:35.966787 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.966807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.966955 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.967856 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.967864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.967896 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:35.968078 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.968086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.969646 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.969651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.971061 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.971071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.972135 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.972139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.972285 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.972345 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.972469 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.972622 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":510,"thread_start_us":142,"threads_started":1}
I20260923 18:05:35.973042 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.965993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.18.44.193:38751)
0923 18:05:35.967297 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.967305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.967335 (+ 30us) server_negotiation.cc:407] Connection header received
0923 18:05:35.967425 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.967430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.967666 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.967858 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:35.968424 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.969496 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.971213 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.971992 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.972105 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.972284 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.972675 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.972855 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":90,"threads_started":1}
I20260923 18:05:35.974053 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.966775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:34459 (local address 127.0.0.1:34998)
0923 18:05:35.968021 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.968038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.968168 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.968881 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.968884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.968903 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:35.969042 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.969049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.970584 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.970587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.972208 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.972231 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.973626 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.973630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.973704 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.973723 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.973795 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.973899 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":103,"threads_started":1}
I20260923 18:05:35.974449 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.967209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.18.44.195:34459)
0923 18:05:35.967664 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.967674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.968279 (+ 605us) server_negotiation.cc:407] Connection header received
0923 18:05:35.968369 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.968374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.968539 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.968695 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:35.969182 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.970419 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.972413 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.973284 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.973400 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.973565 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.973870 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.973984 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":162,"threads_started":1}
I20260923 18:05:35.975335 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.965729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36497 (local address 127.0.0.1:50446)
0923 18:05:35.966709 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:35.966770 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:35.966953 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:35.968679 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:35.968682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:35.968703 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:35.968820 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.968826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.970326 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.970330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.972369 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:35.972377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.974746 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.974751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:35.974885 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.974918 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 18:05:35.975007 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 18:05:35.975115 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":140,"threads_started":1}
I20260923 18:05:35.975742 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:35.966045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.18.44.194:36497)
0923 18:05:35.968014 (+ 1969us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:35.968042 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:35.968077 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:05:35.968195 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:35.968201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:35.968490 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:35.968695 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:35.968992 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.970164 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.972520 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:35.973902 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:35.974071 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:35.975285 (+ 1214us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:35.975383 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 18:05:35.975528 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1168,"thread_start_us":190,"threads_started":1}
I20260923 18:05:35.977396 19863 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 (DEFAULT_TABLE table=test-table [id=1a0d709a5f6043159bb3b814c6a5718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:35.977855 20143 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 (DEFAULT_TABLE table=test-table [id=1a0d709a5f6043159bb3b814c6a5718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:35.978899 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd564a5b5f4701b9cfac78882993e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:35.979365 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd564a5b5f4701b9cfac78882993e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:35.978961 19998 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 (DEFAULT_TABLE table=test-table [id=1a0d709a5f6043159bb3b814c6a5718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:35.980336 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd564a5b5f4701b9cfac78882993e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:35.989423 20232 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Bootstrap starting.
I20260923 18:05:35.991720 20233 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Bootstrap starting.
I20260923 18:05:35.994024 20232 tablet_bootstrap.cc:654] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:35.994439 20233 tablet_bootstrap.cc:654] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:35.994884 20234 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Bootstrap starting.
I20260923 18:05:35.995466 20233 log.cc:824] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:35.995656 20232 log.cc:824] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:35.998183 20232 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: No bootstrap required, opened a new log
I20260923 18:05:35.998459 20232 ts_tablet_manager.cc:1397] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 18:05:35.998450 20234 tablet_bootstrap.cc:654] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:35.998606 20233 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: No bootstrap required, opened a new log
I20260923 18:05:35.998874 20233 ts_tablet_manager.cc:1397] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:05:35.999693 20234 log.cc:824] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:36.002882 20232 raft_consensus.cc:348] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.003201 20232 raft_consensus.cc:374] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.003283 20232 raft_consensus.cc:729] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181cd240b5e543a2ae5dea36712c7909, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.003443 20234 tablet_bootstrap.cc:492] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: No bootstrap required, opened a new log
I20260923 18:05:36.003749 20234 ts_tablet_manager.cc:1397] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 18:05:36.003924 20232 consensus_queue.cc:260] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.003897 20233 raft_consensus.cc:348] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.004289 20233 raft_consensus.cc:374] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.004364 20233 raft_consensus.cc:729] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20dabf8028f4471da81e0651f3cbe229, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.004944 19929 heartbeater.cc:503] Master 127.18.44.254:33047 was elected leader, sending a full tablet report...
I20260923 18:05:36.004983 20233 consensus_queue.cc:260] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.005148 20232 ts_tablet_manager.cc:1428] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 18:05:36.006021 20209 heartbeater.cc:503] Master 127.18.44.254:33047 was elected leader, sending a full tablet report...
I20260923 18:05:36.006547 20233 ts_tablet_manager.cc:1428] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260923 18:05:36.009485 20234 raft_consensus.cc:348] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.009927 20234 raft_consensus.cc:374] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.010023 20234 raft_consensus.cc:729] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b7d6b6af504841a240cd7ab53112e9, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.010607 20234 consensus_queue.cc:260] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.011874 20069 heartbeater.cc:503] Master 127.18.44.254:33047 was elected leader, sending a full tablet report...
I20260923 18:05:36.012122 20234 ts_tablet_manager.cc:1428] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260923 18:05:36.079885 19930 tablet.cc:2369] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:05:36.120379 20070 tablet.cc:2369] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:05:36.152515 20210 tablet.cc:2369] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:05:36.253829 20239 raft_consensus.cc:482] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:36.254041 20239 raft_consensus.cc:504] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.255352 20239 leader_election.cc:288] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497), 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751)
I20260923 18:05:36.260005 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.255837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:36345 (local address 127.18.44.193:38751)
0923 18:05:36.256100 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.256107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.256133 (+ 26us) server_negotiation.cc:407] Connection header received
0923 18:05:36.256226 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.256229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.256296 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.256382 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:36.256726 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.257427 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.258759 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.259298 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.259442 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.259517 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.259726 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.259822 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260923 18:05:36.260022 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.255725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:38751 (local address 127.18.44.195:36345)
0923 18:05:36.255987 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.256004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.256119 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.256464 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.256467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.256483 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:36.256603 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.256608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.257569 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.257573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.258596 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.258610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.259442 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.259446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.259595 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.259620 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.259697 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.259822 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":33}
I20260923 18:05:36.260157 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.255868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:34553 (local address 127.18.44.194:36497)
0923 18:05:36.256103 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.256108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.256398 (+ 290us) server_negotiation.cc:407] Connection header received
0923 18:05:36.256567 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.256570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.256616 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.256693 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:36.257416 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.257952 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.259096 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.259574 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.259697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.259757 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.259939 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.260014 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260923 18:05:36.260231 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.255808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36497 (local address 127.18.44.195:34553)
0923 18:05:36.256299 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.256311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.256417 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.256722 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.256725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.256740 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:36.257276 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.257285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.258091 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.258095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.258964 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.258973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.259698 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.259701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.259817 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.259839 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.259912 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.260003 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":29,"thread_start_us":232,"threads_started":1}
I20260923 18:05:36.260960 20023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd564a5b5f4701b9cfac78882993e3" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b7d6b6af504841a240cd7ab53112e9" is_pre_election: true
I20260923 18:05:36.261023 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd564a5b5f4701b9cfac78882993e3" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909" is_pre_election: true
I20260923 18:05:36.261325 20023 raft_consensus.cc:2454] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 0.
I20260923 18:05:36.261600 19883 raft_consensus.cc:2454] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 0.
I20260923 18:05:36.262009 20094 leader_election.cc:302] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20dabf8028f4471da81e0651f3cbe229, e6b7d6b6af504841a240cd7ab53112e9; no voters:
I20260923 18:05:36.262342 20239 raft_consensus.cc:2781] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:36.262440 20239 raft_consensus.cc:482] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:36.262496 20239 raft_consensus.cc:3037] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.263670 20239 raft_consensus.cc:504] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.264051 20239 leader_election.cc:288] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 election: Requested vote from peers e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497), 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751)
I20260923 18:05:36.264513 20023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd564a5b5f4701b9cfac78882993e3" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b7d6b6af504841a240cd7ab53112e9"
I20260923 18:05:36.264660 20023 raft_consensus.cc:3037] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.264707 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd564a5b5f4701b9cfac78882993e3" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909"
I20260923 18:05:36.264878 19883 raft_consensus.cc:3037] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.265988 20023 raft_consensus.cc:2454] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 1.
I20260923 18:05:36.266180 19883 raft_consensus.cc:2454] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 1.
I20260923 18:05:36.266573 20096 leader_election.cc:302] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181cd240b5e543a2ae5dea36712c7909, 20dabf8028f4471da81e0651f3cbe229; no voters:
I20260923 18:05:36.266855 20239 raft_consensus.cc:2781] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:36.267071 20239 raft_consensus.cc:686] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 1 LEADER]: Becoming Leader. State: Replica: 20dabf8028f4471da81e0651f3cbe229, State: Running, Role: LEADER
I20260923 18:05:36.267560 20239 consensus_queue.cc:237] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.271675 19723 catalog_manager.cc:5495] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } health_report { overall_health: HEALTHY } } }
I20260923 18:05:36.359723 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.354650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.18.44.195:34459)
0923 18:05:36.354831 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.354835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.354876 (+ 41us) server_negotiation.cc:407] Connection header received
0923 18:05:36.355158 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.355161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.355224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.355363 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:36.355738 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.356298 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.357703 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.357987 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.358093 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.358563 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.359097 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.359215 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.359508 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.359577 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:05:36.367486 20023 raft_consensus.cc:1261] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Refusing update from remote peer 20dabf8028f4471da81e0651f3cbe229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.367630 19883 raft_consensus.cc:1261] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Refusing update from remote peer 20dabf8028f4471da81e0651f3cbe229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.368368 20242 consensus_queue.cc:1035] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.368623 20239 consensus_queue.cc:1035] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.396415 19723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:36.396659 19723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:36.400152 19723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Sending DeleteTablet for 3 replicas of tablet dbfd564a5b5f4701b9cfac78882993e3
I20260923 18:05:36.401253 19998 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:50446
I20260923 18:05:36.401404 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40590
I20260923 18:05:36.401520 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfd564a5b5f4701b9cfac78882993e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:34998
I20260923 18:05:36.402151 20256 tablet_replica.cc:330] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: stopping tablet replica
I20260923 18:05:36.402412 20255 tablet_replica.cc:330] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: stopping tablet replica
I20260923 18:05:36.402518 20257 tablet_replica.cc:330] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: stopping tablet replica
I20260923 18:05:36.402988 20257 raft_consensus.cc:2229] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:36.403004 20255 raft_consensus.cc:2229] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:36.403477 20257 raft_consensus.cc:2258] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.403712 20255 raft_consensus.cc:2258] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.403651 19723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:05:36.404237 20256 raft_consensus.cc:2229] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 1 LEADER]: Raft consensus shutting down.
W20260923 18:05:36.404268 19723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:36.404670 20256 raft_consensus.cc:2258] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.406454 20256 ts_tablet_manager.cc:1905] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.407688 20255 ts_tablet_manager.cc:1905] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.408125 20257 ts_tablet_manager.cc:1905] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.410456 20256 ts_tablet_manager.cc:1918] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:36.410604 20256 log.cc:1196] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/dbfd564a5b5f4701b9cfac78882993e3
I20260923 18:05:36.410912 20256 ts_tablet_manager.cc:1939] T dbfd564a5b5f4701b9cfac78882993e3 P 20dabf8028f4471da81e0651f3cbe229: Deleting consensus metadata
I20260923 18:05:36.412801 19863 tablet_service.cc:1467] Processing CreateTablet for tablet 4361f0467dce4aacad3291055fce4b27 (DEFAULT_TABLE table=test-table [id=9f92bc33625e4235b9522413b6b09431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:36.412879 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 4361f0467dce4aacad3291055fce4b27 (DEFAULT_TABLE table=test-table [id=9f92bc33625e4235b9522413b6b09431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:36.413188 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4361f0467dce4aacad3291055fce4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.413188 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4361f0467dce4aacad3291055fce4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.416417 19711 catalog_manager.cc:4841] TS 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195:34459): tablet dbfd564a5b5f4701b9cfac78882993e3 (table test-table [id=1a0d709a5f6043159bb3b814c6a5718b]) successfully deleted
I20260923 18:05:36.416678 20233 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Bootstrap starting.
I20260923 18:05:36.417558 19998 tablet_service.cc:1467] Processing CreateTablet for tablet 4361f0467dce4aacad3291055fce4b27 (DEFAULT_TABLE table=test-table [id=9f92bc33625e4235b9522413b6b09431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:36.418120 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4361f0467dce4aacad3291055fce4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.418107 20233 tablet_bootstrap.cc:654] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.419459 20257 ts_tablet_manager.cc:1918] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:36.420260 20255 ts_tablet_manager.cc:1918] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:36.420420 20255 log.cc:1196] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/dbfd564a5b5f4701b9cfac78882993e3
I20260923 18:05:36.420763 20233 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: No bootstrap required, opened a new log
I20260923 18:05:36.420905 20233 ts_tablet_manager.cc:1397] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:36.420945 20257 log.cc:1196] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/dbfd564a5b5f4701b9cfac78882993e3
I20260923 18:05:36.421309 20257 ts_tablet_manager.cc:1939] T dbfd564a5b5f4701b9cfac78882993e3 P 181cd240b5e543a2ae5dea36712c7909: Deleting consensus metadata
I20260923 18:05:36.420830 20255 ts_tablet_manager.cc:1939] T dbfd564a5b5f4701b9cfac78882993e3 P e6b7d6b6af504841a240cd7ab53112e9: Deleting consensus metadata
I20260923 18:05:36.421564 20233 raft_consensus.cc:348] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.421739 20233 raft_consensus.cc:374] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.421784 20233 raft_consensus.cc:729] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20dabf8028f4471da81e0651f3cbe229, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.421942 20233 consensus_queue.cc:260] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.422533 20233 ts_tablet_manager.cc:1428] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:05:36.423961 20232 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Bootstrap starting.
I20260923 18:05:36.424428 19712 catalog_manager.cc:4841] TS 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751): tablet dbfd564a5b5f4701b9cfac78882993e3 (table test-table [id=1a0d709a5f6043159bb3b814c6a5718b]) successfully deleted
I20260923 18:05:36.425230 19710 catalog_manager.cc:4841] TS e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497): tablet dbfd564a5b5f4701b9cfac78882993e3 (table test-table [id=1a0d709a5f6043159bb3b814c6a5718b]) successfully deleted
I20260923 18:05:36.425503 20234 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Bootstrap starting.
I20260923 18:05:36.425755 20232 tablet_bootstrap.cc:654] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.426728 20234 tablet_bootstrap.cc:654] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.429096 20234 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: No bootstrap required, opened a new log
I20260923 18:05:36.429235 20234 ts_tablet_manager.cc:1397] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:36.429351 20232 tablet_bootstrap.cc:492] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: No bootstrap required, opened a new log
I20260923 18:05:36.429563 20232 ts_tablet_manager.cc:1397] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 18:05:36.429844 20234 raft_consensus.cc:348] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.429997 20234 raft_consensus.cc:374] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.430030 20234 raft_consensus.cc:729] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b7d6b6af504841a240cd7ab53112e9, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.430181 20234 consensus_queue.cc:260] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.430153 20232 raft_consensus.cc:348] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.430289 20232 raft_consensus.cc:374] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.430322 20232 raft_consensus.cc:729] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181cd240b5e543a2ae5dea36712c7909, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.430466 20232 consensus_queue.cc:260] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.430505 20234 ts_tablet_manager.cc:1428] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 18:05:36.430775 20232 ts_tablet_manager.cc:1428] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:05:36.584197 20240 raft_consensus.cc:482] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:36.584416 20240 raft_consensus.cc:504] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.585908 20240 leader_election.cc:288] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751), 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195:34459)
I20260923 18:05:36.590948 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.586448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:41073 (local address 127.18.44.193:38751)
0923 18:05:36.586646 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.586651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.586844 (+ 193us) server_negotiation.cc:407] Connection header received
0923 18:05:36.587197 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.587201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.587279 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.587349 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:36.587939 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.588591 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.589894 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.590346 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.590474 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.590536 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.590726 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.590808 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:05:36.590956 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.586500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:44087 (local address 127.18.44.195:34459)
0923 18:05:36.586869 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.586874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.586895 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:05:36.587157 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.587161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.587227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.587326 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:36.587764 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.588387 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.589745 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.590230 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.590344 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.590402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.590684 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.590786 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260923 18:05:36.591041 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.586367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:34459 (local address 127.18.44.194:44087)
0923 18:05:36.586808 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.586823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.586995 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.587414 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.587418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.587449 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:36.587616 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.587624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.588528 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.588532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.589598 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.589609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.590394 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.590398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.590511 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.590534 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.590614 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.590698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"spinlock_wait_cycles":16384,"thread_start_us":160,"threads_started":1}
I20260923 18:05:36.591045 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.586232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:38751 (local address 127.18.44.194:41073)
0923 18:05:36.586711 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.586729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.586994 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.587475 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.587485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.587510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:36.587779 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.587787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.588715 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.588719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.589761 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.589772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.590476 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.590485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.590581 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.590607 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.590693 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.590786 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":58880,"thread_start_us":99,"threads_started":1}
I20260923 18:05:36.591784 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4361f0467dce4aacad3291055fce4b27" candidate_uuid: "e6b7d6b6af504841a240cd7ab53112e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909" is_pre_election: true
I20260923 18:05:36.591777 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4361f0467dce4aacad3291055fce4b27" candidate_uuid: "e6b7d6b6af504841a240cd7ab53112e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20dabf8028f4471da81e0651f3cbe229" is_pre_election: true
I20260923 18:05:36.591986 19883 raft_consensus.cc:2454] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b7d6b6af504841a240cd7ab53112e9 in term 0.
I20260923 18:05:36.592051 20163 raft_consensus.cc:2454] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b7d6b6af504841a240cd7ab53112e9 in term 0.
I20260923 18:05:36.592638 19955 leader_election.cc:302] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20dabf8028f4471da81e0651f3cbe229, e6b7d6b6af504841a240cd7ab53112e9; no voters:
I20260923 18:05:36.593137 20240 raft_consensus.cc:2781] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:36.593237 20240 raft_consensus.cc:482] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:36.593274 20240 raft_consensus.cc:3037] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.594767 20240 raft_consensus.cc:504] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.595716 20240 leader_election.cc:288] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [CANDIDATE]: Term 1 election: Requested vote from peers 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751), 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195:34459)
I20260923 18:05:36.596266 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4361f0467dce4aacad3291055fce4b27" candidate_uuid: "e6b7d6b6af504841a240cd7ab53112e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909"
I20260923 18:05:36.596316 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4361f0467dce4aacad3291055fce4b27" candidate_uuid: "e6b7d6b6af504841a240cd7ab53112e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20dabf8028f4471da81e0651f3cbe229"
I20260923 18:05:36.596448 19883 raft_consensus.cc:3037] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.596473 20163 raft_consensus.cc:3037] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.597705 20163 raft_consensus.cc:2454] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b7d6b6af504841a240cd7ab53112e9 in term 1.
I20260923 18:05:36.597914 19883 raft_consensus.cc:2454] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b7d6b6af504841a240cd7ab53112e9 in term 1.
I20260923 18:05:36.598155 19955 leader_election.cc:302] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20dabf8028f4471da81e0651f3cbe229, e6b7d6b6af504841a240cd7ab53112e9; no voters:
I20260923 18:05:36.598446 20240 raft_consensus.cc:2781] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:36.598759 20240 raft_consensus.cc:686] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 LEADER]: Becoming Leader. State: Replica: e6b7d6b6af504841a240cd7ab53112e9, State: Running, Role: LEADER
I20260923 18:05:36.599076 20240 consensus_queue.cc:237] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.601799 19723 catalog_manager.cc:5495] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "e6b7d6b6af504841a240cd7ab53112e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } health_report { overall_health: UNKNOWN } } }
Sep 23 18:05:36 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186734, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:36.661422 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.645562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33047 (local address 127.0.0.1:32936)
0923 18:05:36.646018 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.646038 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.646316 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.646887 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.646890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.647713 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:36.648043 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.648052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.649549 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.649554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.650582 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.650600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.651472 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.651475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.651726 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.652592 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:36.652607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:36.655032 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:36.658603 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:36.658623 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:36.658630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:36.659015 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:36.659593 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:36.659596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:36.659597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:36.659861 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:36.660747 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:36.660754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:36.660931 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.661076 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.661254 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":176,"threads_started":1}
I20260923 18:05:36.662119 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.645744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.18.44.254:33047)
0923 18:05:36.646285 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.646298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.646321 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:05:36.646491 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.646494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.646551 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.646673 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:36.648434 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.649229 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.650800 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.651074 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.651330 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.655528 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:36.655536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:36.655539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:36.655593 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:36.658247 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.659199 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.659203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.659204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.659422 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.660027 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.660031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.660032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.660503 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:36.660644 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.661190 (+ 546us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.661879 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260923 18:05:36.668282 19723 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:32936:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:05:36.669202 19723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:36.678989 19998 tablet_service.cc:1467] Processing CreateTablet for tablet c80cdabb85cf49e5bf62f4b09d401ac4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1dbb1d34b164ad7b76e7cdc4045f7b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:36.678989 19863 tablet_service.cc:1467] Processing CreateTablet for tablet c80cdabb85cf49e5bf62f4b09d401ac4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1dbb1d34b164ad7b76e7cdc4045f7b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:36.679057 20143 tablet_service.cc:1467] Processing CreateTablet for tablet c80cdabb85cf49e5bf62f4b09d401ac4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1dbb1d34b164ad7b76e7cdc4045f7b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:36.679637 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80cdabb85cf49e5bf62f4b09d401ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.679656 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80cdabb85cf49e5bf62f4b09d401ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.679656 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80cdabb85cf49e5bf62f4b09d401ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:36.683833 20233 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: Bootstrap starting.
I20260923 18:05:36.684387 20234 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: Bootstrap starting.
I20260923 18:05:36.684991 20233 tablet_bootstrap.cc:654] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.685002 20232 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: Bootstrap starting.
I20260923 18:05:36.686074 20234 tablet_bootstrap.cc:654] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.686385 20232 tablet_bootstrap.cc:654] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:36.687047 20233 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: No bootstrap required, opened a new log
I20260923 18:05:36.687162 20233 ts_tablet_manager.cc:1397] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:36.688076 20233 raft_consensus.cc:348] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.688222 20233 raft_consensus.cc:374] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.688474 20233 raft_consensus.cc:729] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20dabf8028f4471da81e0651f3cbe229, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.688755 20232 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: No bootstrap required, opened a new log
I20260923 18:05:36.688712 20233 consensus_queue.cc:260] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.688910 20232 ts_tablet_manager.cc:1397] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:36.689189 20252 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.689532 20252 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.689615 20234 tablet_bootstrap.cc:492] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: No bootstrap required, opened a new log
I20260923 18:05:36.689715 20234 ts_tablet_manager.cc:1397] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 18:05:36.689900 20232 raft_consensus.cc:348] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.690402 20232 raft_consensus.cc:374] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.690488 20232 raft_consensus.cc:729] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181cd240b5e543a2ae5dea36712c7909, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.690575 20234 raft_consensus.cc:348] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.690727 20234 raft_consensus.cc:374] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:36.690768 20234 raft_consensus.cc:729] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b7d6b6af504841a240cd7ab53112e9, State: Initialized, Role: FOLLOWER
I20260923 18:05:36.690927 20234 consensus_queue.cc:260] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.691170 20240 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.691360 20240 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.691215 20232 consensus_queue.cc:260] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.690609 20233 ts_tablet_manager.cc:1428] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 18:05:36.691624 20234 ts_tablet_manager.cc:1428] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:05:36.691655 20232 ts_tablet_manager.cc:1428] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:36.691715 20238 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.691892 20238 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.718806 20252 raft_consensus.cc:482] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:36.718972 20252 raft_consensus.cc:504] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.719662 20252 leader_election.cc:288] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751), e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497)
I20260923 18:05:36.720139 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80cdabb85cf49e5bf62f4b09d401ac4" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909" is_pre_election: true
I20260923 18:05:36.720198 20023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80cdabb85cf49e5bf62f4b09d401ac4" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b7d6b6af504841a240cd7ab53112e9" is_pre_election: true
I20260923 18:05:36.720324 20023 raft_consensus.cc:2454] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 0.
I20260923 18:05:36.720324 19883 raft_consensus.cc:2454] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 0.
I20260923 18:05:36.720696 20096 leader_election.cc:302] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181cd240b5e543a2ae5dea36712c7909, 20dabf8028f4471da81e0651f3cbe229; no voters:
I20260923 18:05:36.721171 20252 raft_consensus.cc:2781] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:36.721256 20252 raft_consensus.cc:482] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:36.721287 20252 raft_consensus.cc:3037] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.722417 20252 raft_consensus.cc:504] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.722844 20252 leader_election.cc:288] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 election: Requested vote from peers 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751), e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497)
I20260923 18:05:36.723912 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80cdabb85cf49e5bf62f4b09d401ac4" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181cd240b5e543a2ae5dea36712c7909"
I20260923 18:05:36.723912 20023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80cdabb85cf49e5bf62f4b09d401ac4" candidate_uuid: "20dabf8028f4471da81e0651f3cbe229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b7d6b6af504841a240cd7ab53112e9"
I20260923 18:05:36.724402 19883 raft_consensus.cc:3037] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.724428 20023 raft_consensus.cc:3037] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:36.726579 19883 raft_consensus.cc:2454] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 1.
I20260923 18:05:36.726613 20023 raft_consensus.cc:2454] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dabf8028f4471da81e0651f3cbe229 in term 1.
I20260923 18:05:36.727191 20096 leader_election.cc:302] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181cd240b5e543a2ae5dea36712c7909, 20dabf8028f4471da81e0651f3cbe229; no voters:
I20260923 18:05:36.727674 20252 raft_consensus.cc:2781] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:36.727799 20252 raft_consensus.cc:686] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [term 1 LEADER]: Becoming Leader. State: Replica: 20dabf8028f4471da81e0651f3cbe229, State: Running, Role: LEADER
I20260923 18:05:36.727962 20252 consensus_queue.cc:237] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } }
I20260923 18:05:36.728799 20242 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: TxnStatusTablet state changed. Reason: New leader 20dabf8028f4471da81e0651f3cbe229. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.729001 20242 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:36.729553 20276 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:05:36.729756 20276 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:05:36.730111 19725 catalog_manager.cc:5495] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } health_report { overall_health: HEALTHY } } }
I20260923 18:05:36.757231 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.752978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:34459 (local address 127.0.0.1:35014)
0923 18:05:36.753191 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.753210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.753372 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.753721 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.753724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.753732 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:36.753865 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.753870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.754665 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.754669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.755417 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.755430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.755887 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.755891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.755992 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.756126 (+ 134us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.756468 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.756487 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.756760 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.757093 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 18:05:36.757220 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.753164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.18.44.195:34459)
0923 18:05:36.753354 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.753360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.753381 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:05:36.753550 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.753552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.753608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.753678 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:36.753967 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.754545 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.755540 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.755767 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.755863 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.756292 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.756392 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.756452 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.756781 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.757019 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 18:05:36.761626 20023 raft_consensus.cc:1261] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Refusing update from remote peer 20dabf8028f4471da81e0651f3cbe229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.761703 19883 raft_consensus.cc:1261] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Refusing update from remote peer 20dabf8028f4471da81e0651f3cbe229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.762290 20252 consensus_queue.cc:1035] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.762605 20242 consensus_queue.cc:1035] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.764902 20240 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: TxnStatusTablet state changed. Reason: New leader 20dabf8028f4471da81e0651f3cbe229. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.765115 20240 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.766011 20238 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: TxnStatusTablet state changed. Reason: New leader 20dabf8028f4471da81e0651f3cbe229. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.766202 20238 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.767882 20242 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.768093 20242 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:36.768210 20245 mvcc.cc:205] Tried to move back new op lower bound from 7332604873767895040 to 7332604873639280640. Current Snapshot: MvccSnapshot[applied={T|T < 7332604873767895040 or (T in {7332604873767895040})}]
I20260923 18:05:36.768774 20246 mvcc.cc:205] Tried to move back new op lower bound from 7332604873767895040 to 7332604873639280640. Current Snapshot: MvccSnapshot[applied={T|T < 7332604873767895040 or (T in {7332604873767895040})}]
I20260923 18:05:36.769615 20238 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.769770 20238 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 181cd240b5e543a2ae5dea36712c7909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.772842 20242 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.773092 20242 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P 20dabf8028f4471da81e0651f3cbe229: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:36.773808 20240 tablet_replica.cc:439] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20dabf8028f4471da81e0651f3cbe229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 } } peers { permanent_uuid: "e6b7d6b6af504841a240cd7ab53112e9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36497 } } peers { permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 } } }
I20260923 18:05:36.773979 20240 tablet_replica.cc:442] T c80cdabb85cf49e5bf62f4b09d401ac4 P e6b7d6b6af504841a240cd7ab53112e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:36.774425 20247 mvcc.cc:205] Tried to move back new op lower bound from 7332604873767895040 to 7332604873639280640. Current Snapshot: MvccSnapshot[applied={T|T < 7332604873767895040 or (T in {7332604873767895040})}]
I20260923 18:05:36.786936 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.779371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.18.44.194:36497)
0923 18:05:36.779841 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.779850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.779871 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:05:36.779981 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.779984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.780061 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.780483 (+ 422us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:36.780865 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.781774 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.783084 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.783584 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.783911 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.785019 (+ 1108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.785739 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.786598 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.786643 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.786737 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":203,"threads_started":1}
I20260923 18:05:36.804669 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.799580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:51251 (local address 127.18.44.195:34459)
0923 18:05:36.799826 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.799831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.800087 (+ 256us) server_negotiation.cc:407] Connection header received
0923 18:05:36.800252 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.800256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.800323 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.800459 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:36.800914 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.801826 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.802909 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.803289 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.803447 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.804025 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.804138 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.804227 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.804425 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.804495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 18:05:36.804752 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.799453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:34459 (local address 127.18.44.194:51251)
0923 18:05:36.799964 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.799980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.800130 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.800518 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.800522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.800532 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:36.800726 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.800733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.802054 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.802058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.802763 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.802775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.803582 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.803586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.803724 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.803818 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.804301 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.804311 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.804392 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.804540 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":174,"threads_started":1}
I20260923 18:05:36.816231 20163 raft_consensus.cc:1261] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Refusing update from remote peer e6b7d6b6af504841a240cd7ab53112e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.817313 19883 raft_consensus.cc:1261] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Refusing update from remote peer e6b7d6b6af504841a240cd7ab53112e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:36.817626 20240 consensus_queue.cc:1035] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20dabf8028f4471da81e0651f3cbe229" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.818614 20240 consensus_queue.cc:1035] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181cd240b5e543a2ae5dea36712c7909" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:36.855578 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.849866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36497 (local address 127.18.44.195:58031)
0923 18:05:36.850541 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:36.850560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:36.850734 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:36.851115 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:36.851117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:36.851125 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:36.851244 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.851286 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.852399 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.852402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.853629 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:36.853641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.854361 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.854365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:36.854466 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.854542 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.855149 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.855159 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 18:05:36.855226 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 18:05:36.855402 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":197,"threads_started":1}
I20260923 18:05:36.855655 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.850066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:58031 (local address 127.18.44.194:36497)
0923 18:05:36.850243 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.850248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.850643 (+ 395us) server_negotiation.cc:407] Connection header received
0923 18:05:36.850862 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.850865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.850921 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.850968 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:36.851404 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.852225 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.853832 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.854182 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.854307 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.854817 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:36.854954 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:36.855050 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.855298 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.855380 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:05:36.891419 19725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:36.891616 19725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:36.894264 19725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4b8ac97d83b402798c8daa0d6b94734: Sending DeleteTablet for 3 replicas of tablet 4361f0467dce4aacad3291055fce4b27
I20260923 18:05:36.895455 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet 4361f0467dce4aacad3291055fce4b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40590
I20260923 18:05:36.895823 20257 tablet_replica.cc:330] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: stopping tablet replica
I20260923 18:05:36.895823 19998 tablet_service.cc:1514] Processing DeleteTablet for tablet 4361f0467dce4aacad3291055fce4b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:50446
I20260923 18:05:36.895980 20257 raft_consensus.cc:2229] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:36.896081 20255 tablet_replica.cc:330] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: stopping tablet replica
I20260923 18:05:36.896140 20257 raft_consensus.cc:2258] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.896221 20255 raft_consensus.cc:2229] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:36.896661 20255 raft_consensus.cc:2258] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.898331 20257 ts_tablet_manager.cc:1905] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.902738 20257 ts_tablet_manager.cc:1918] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:36.902892 20257 log.cc:1196] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/4361f0467dce4aacad3291055fce4b27
I20260923 18:05:36.903167 20257 ts_tablet_manager.cc:1939] T 4361f0467dce4aacad3291055fce4b27 P 181cd240b5e543a2ae5dea36712c7909: Deleting consensus metadata
I20260923 18:05:36.904309 20255 ts_tablet_manager.cc:1905] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.907474 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet 4361f0467dce4aacad3291055fce4b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:34998
I20260923 18:05:36.907765 20256 tablet_replica.cc:330] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: stopping tablet replica
I20260923 18:05:36.907918 20256 raft_consensus.cc:2229] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:36.908105 20256 raft_consensus.cc:2258] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:36.909157 20256 ts_tablet_manager.cc:1905] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:36.912829 20255 ts_tablet_manager.cc:1918] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:36.912952 20255 log.cc:1196] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/4361f0467dce4aacad3291055fce4b27
I20260923 18:05:36.913184 20256 ts_tablet_manager.cc:1918] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:36.913233 20255 ts_tablet_manager.cc:1939] T 4361f0467dce4aacad3291055fce4b27 P e6b7d6b6af504841a240cd7ab53112e9: Deleting consensus metadata
I20260923 18:05:36.913301 20256 log.cc:1196] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/4361f0467dce4aacad3291055fce4b27
I20260923 18:05:36.913578 20256 ts_tablet_manager.cc:1939] T 4361f0467dce4aacad3291055fce4b27 P 20dabf8028f4471da81e0651f3cbe229: Deleting consensus metadata
I20260923 18:05:36.914240 19710 catalog_manager.cc:4841] TS e6b7d6b6af504841a240cd7ab53112e9 (127.18.44.194:36497): tablet 4361f0467dce4aacad3291055fce4b27 (table test-table [id=9f92bc33625e4235b9522413b6b09431]) successfully deleted
I20260923 18:05:36.914680 19711 catalog_manager.cc:4841] TS 20dabf8028f4471da81e0651f3cbe229 (127.18.44.195:34459): tablet 4361f0467dce4aacad3291055fce4b27 (table test-table [id=9f92bc33625e4235b9522413b6b09431]) successfully deleted
I20260923 18:05:36.914940 19712 catalog_manager.cc:4841] TS 181cd240b5e543a2ae5dea36712c7909 (127.18.44.193:38751): tablet 4361f0467dce4aacad3291055fce4b27 (table test-table [id=9f92bc33625e4235b9522413b6b09431]) successfully deleted
Sep 23 18:05:36 dist-test-slave-rvjj krb5kdc[19681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:36.930969 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.915146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.18.44.193:38751)
0923 18:05:36.915340 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.915345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.915928 (+ 583us) server_negotiation.cc:407] Connection header received
0923 18:05:36.916133 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.916139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.916218 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.916499 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:36.917346 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.918266 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.919634 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.920043 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.920174 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.925408 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:36.925425 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:36.925447 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:36.925490 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:36.927990 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.929016 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.929024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.929026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.929162 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.929620 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.929623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.929625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.929997 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:36.930236 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.930502 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.930716 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 18:05:36.931764 19923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40600
I20260923 18:05:36.945080 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:36.934497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.18.44.254:33047)
0923 18:05:36.934703 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:36.934709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:36.934735 (+ 26us) server_negotiation.cc:407] Connection header received
0923 18:05:36.934917 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:36.934920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:36.934987 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:36.935093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:36.935946 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.936866 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.937875 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:36.938272 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:36.938393 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:36.940847 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:36.940852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:36.940854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:36.940900 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:36.942792 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.943593 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.943597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.943598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.943704 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:36.944090 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:36.944093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:36.944094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:36.944338 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:36.944460 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:36.944799 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 18:05:36.944944 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 18:05:36.945591 19725 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32952
I20260923 18:05:36.947578 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19798
I20260923 18:05:36.955438 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19938
I20260923 18:05:36.965968 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 20078
I20260923 18:05:36.973680 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 19694
2026-09-23T18:05:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3455 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:37 dist-test-slave-rvjj krb5kdc[20298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:37 dist-test-slave-rvjj krb5kdc[20298](info): set up 2 sockets
Sep 23 18:05:37 dist-test-slave-rvjj krb5kdc[20298](info): commencing operation
krb5kdc: starting...
W20260923 18:05:37.992190 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:38.136783 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37413
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:38.267184 20311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:38.267553 20311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:38.267596 20311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:38.272109 20311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:38.272305 20311 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:05:38.272361 20311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:38.272404 20311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:38.272444 20311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:38.272464 20311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:38.279316 20311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37413
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:38.280519 20311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:38.281615 20311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:38.290994 20316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.291069 20319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.291186 20317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:38.291471 20311 server_base.cc:1034] running on GCE node
I20260923 18:05:38.292115 20311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:38.293388 20311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:38.294612 20311 hybrid_clock.cc:648] HybridClock initialized: now 1790186738294544 us; error 84 us; skew 500 ppm
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:38.302466 20311 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:38.303750 20311 webserver.cc:460] Webserver started at http://127.18.44.254:36239/ using document root <none> and password file <none>
I20260923 18:05:38.304401 20311 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:38.304500 20311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:38.304723 20311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:38.306973 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "bd8474f5d9b74be38832613a08440af6"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "31db7c673d3de1d1a3393adb6010df65"
server_key_iv: "1227d29d9a285f52c59a2b3d70102a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.307642 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "bd8474f5d9b74be38832613a08440af6"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "31db7c673d3de1d1a3393adb6010df65"
server_key_iv: "1227d29d9a285f52c59a2b3d70102a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.312340 20311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:05:38.316100 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:38.318017 20311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260923 18:05:38.318210 20311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "bd8474f5d9b74be38832613a08440af6"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "31db7c673d3de1d1a3393adb6010df65"
server_key_iv: "1227d29d9a285f52c59a2b3d70102a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.318349 20311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:38.345657 20311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:38.350736 20311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:38.350970 20311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:38.360991 20311 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37413
I20260923 18:05:38.361042 20388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37413 every 8 connection(s)
I20260923 18:05:38.362102 20311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:38.365909 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:38.366142 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 20311
I20260923 18:05:38.366262 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:38.366554 18611 external_mini_cluster.cc:1428] Setting key 1bf1564d1717cbfb891310f14a3af54f
I20260923 18:05:38.373580 20389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Bootstrap starting.
I20260923 18:05:38.376560 20389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Neither blocks nor log segments found. Creating new log.
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:38.377801 20389 log.cc:824] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:38.380378 20389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: No bootstrap required, opened a new log
I20260923 18:05:38.384176 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:38.368232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.18.44.254:37413)
0923 18:05:38.368767 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:38.368783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:38.368825 (+ 42us) server_negotiation.cc:407] Connection header received
0923 18:05:38.369738 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:38.369765 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:38.370162 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:38.370552 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:38.371542 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.372604 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.373544 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.374315 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.374527 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:38.377870 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:38.377886 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:38.377904 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:38.377973 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:38.380664 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.381223 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.381229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.381236 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.381379 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.381675 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.381677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.381679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.382163 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:38.382381 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:38.382889 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 18:05:38.383223 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":156,"threads_started":1}
I20260923 18:05:38.384126 20389 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } }
I20260923 18:05:38.384483 20389 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:38.384698 20389 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8474f5d9b74be38832613a08440af6, State: Initialized, Role: FOLLOWER
I20260923 18:05:38.385257 20389 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } }
I20260923 18:05:38.385452 20389 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:38.385517 20389 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:38.385618 20389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:38.386693 20389 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } }
I20260923 18:05:38.387456 20389 leader_election.cc:302] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd8474f5d9b74be38832613a08440af6; no voters:
I20260923 18:05:38.387830 20389 leader_election.cc:288] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:38.388007 20394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:38.388307 20394 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [term 1 LEADER]: Becoming Leader. State: Replica: bd8474f5d9b74be38832613a08440af6, State: Running, Role: LEADER
I20260923 18:05:38.388644 20394 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } }
I20260923 18:05:38.389338 20389 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:38.390547 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8474f5d9b74be38832613a08440af6. Latest consensus state: current_term: 1 leader_uuid: "bd8474f5d9b74be38832613a08440af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } } }
I20260923 18:05:38.390727 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:38.391014 20395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd8474f5d9b74be38832613a08440af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8474f5d9b74be38832613a08440af6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37413 } } }
I20260923 18:05:38.391108 20395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6 [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:38.396220 20409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:38.396463 20409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:38.397073 20403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:38.397980 20403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:38.405452 20403 catalog_manager.cc:1347] Generated new cluster ID: 6642cd0a375849829a7b6b4a2af1f421
I20260923 18:05:38.405541 20403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:38.411859 20403 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:38.412890 20403 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:38.419566 20403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Generated new TSK 0
I20260923 18:05:38.420588 20403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:38.472702 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37413
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:38.624415 20415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:38.624744 20415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:38.624795 20415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:38.630031 20415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:38.630147 20415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:38.630191 20415 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:05:38.630205 20415 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:38.630239 20415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:38.636430 20415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:38.637630 20415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:38.638742 20415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:38.650064 20421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.650069 20420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.650076 20423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:38.650490 20415 server_base.cc:1034] running on GCE node
I20260923 18:05:38.651136 20415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:38.651952 20415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:38.655385 20415 hybrid_clock.cc:648] HybridClock initialized: now 1790186738655343 us; error 121 us; skew 500 ppm
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:38.659323 20415 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:38.660491 20415 webserver.cc:460] Webserver started at http://127.18.44.193:41125/ using document root <none> and password file <none>
I20260923 18:05:38.661123 20415 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:38.661198 20415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:38.661434 20415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:38.663843 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "e2293405e3e90c22dce15d07e1e9303b"
server_key_iv: "abf1b549528eb65831f172bafc9adbf5"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.664487 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "e2293405e3e90c22dce15d07e1e9303b"
server_key_iv: "abf1b549528eb65831f172bafc9adbf5"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.669380 20415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:38.673040 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:38.675015 20415 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:38.675201 20415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "e2293405e3e90c22dce15d07e1e9303b"
server_key_iv: "abf1b549528eb65831f172bafc9adbf5"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.675400 20415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:38.691040 20415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:38.694648 20415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:38.694839 20415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:38.696657 20415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:38.696734 20415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:38.696789 20415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:38.696810 20415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:38.708194 20415 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:33869
I20260923 18:05:38.709362 20415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:38.708230 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:33869 every 8 connection(s)
I20260923 18:05:38.713992 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 20415
I20260923 18:05:38.714309 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:38.714705 18611 external_mini_cluster.cc:1428] Setting key c8031e2fc9c32608f6cb772dcbc31a11
I20260923 18:05:38.718879 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:38.698663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.193:37507)
0923 18:05:38.699406 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:38.699477 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:38.700796 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:38.701748 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:38.701759 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:38.702283 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:38.703169 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:38.703193 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.704503 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.704508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:38.705904 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:38.705923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.706718 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.706723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:38.707014 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:38.708050 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:38.708077 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:38.710668 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:38.713963 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:38.713972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:38.713996 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:38.714474 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:38.714992 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:38.714995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:38.714997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:38.715308 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:38.716035 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:38.716046 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:38.716436 (+ 390us) client_negotiation.cc:769] Sending connection context
0923 18:05:38.716681 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 18:05:38.717185 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":200,"threads_started":1}
I20260923 18:05:38.719919 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:38.697903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:37507 (local address 127.18.44.254:37413)
0923 18:05:38.698133 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:38.698139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:38.699665 (+ 1526us) server_negotiation.cc:407] Connection header received
0923 18:05:38.701199 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:38.701204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:38.701291 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:38.701445 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:38.703486 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.704322 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.706097 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.706465 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.706607 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:38.711033 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:38.711038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:38.711041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:38.711080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:38.713723 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.714628 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.714632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.714634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.714768 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.715463 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.715468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.715470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.715881 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:38.716021 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:38.719586 (+ 3565us) server_negotiation.cc:299] Negotiation successful
0923 18:05:38.719766 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:05:38.721797 20546 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37413
I20260923 18:05:38.722101 20546 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:38.723096 20546 heartbeater.cc:511] Master 127.18.44.254:37413 requested a full tablet report, sending...
I20260923 18:05:38.725714 20342 ts_manager.cc:194] Registered new tserver with Master: fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869)
I20260923 18:05:38.727490 20342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:37507
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:38.736788 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:38.726466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.193:42657)
0923 18:05:38.726961 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:38.726976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:38.727101 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:38.727529 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:38.727532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:38.727798 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:38.727954 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:38.727962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.728983 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.728986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:38.729767 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:38.729825 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.730439 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.730443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:38.730537 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:38.731310 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:38.731327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:38.731825 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:38.734646 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:38.734649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:38.734650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:38.734985 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:38.735461 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:38.735464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:38.735465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:38.735568 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:38.736142 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:38.736144 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:38.736275 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 18:05:38.736390 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 18:05:38.736557 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":134,"threads_started":1}
I20260923 18:05:38.737085 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:38.726624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:42657 (local address 127.18.44.254:37413)
0923 18:05:38.726806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:38.726813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:38.727079 (+ 266us) server_negotiation.cc:407] Connection header received
0923 18:05:38.727222 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:38.727226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:38.727334 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:38.727468 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:38.728129 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.728815 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.729956 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:38.730275 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:38.730400 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:38.732086 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:38.732090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:38.732093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:38.732128 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:38.734371 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.735125 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.735129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.735131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.735302 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:38.735687 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:38.735690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:38.735692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:38.736001 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:38.736118 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:38.736488 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 18:05:38.736855 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:38.757386 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37413
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:38.929739 20555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:38.930179 20555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:38.930238 20555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:38.935568 20555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:38.935702 20555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:38.935757 20555 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:05:38.935771 20555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:38.935817 20555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:38.943226 20555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:38.944952 20555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:38.946628 20555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:38.959383 20563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.959383 20561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:38.959437 20560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:38.959955 20555 server_base.cc:1034] running on GCE node
I20260923 18:05:38.960590 20555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:38.961581 20555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:38.962780 20555 hybrid_clock.cc:648] HybridClock initialized: now 1790186738962741 us; error 65 us; skew 500 ppm
Sep 23 18:05:38 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:38.966706 20555 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:38.968228 20555 webserver.cc:460] Webserver started at http://127.18.44.194:37531/ using document root <none> and password file <none>
I20260923 18:05:38.969036 20555 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:38.969132 20555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:38.969393 20555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:38.972714 20555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "1332353418394447a3543a8890e38e52"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "9a8af5c284b825635fb99430781d6d05"
server_key_iv: "8037d576b99eb234a0ecdd9e689ea28c"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.973544 20555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "1332353418394447a3543a8890e38e52"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "9a8af5c284b825635fb99430781d6d05"
server_key_iv: "8037d576b99eb234a0ecdd9e689ea28c"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.980415 20555 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:05:38.984797 20570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:38.986907 20555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:05:38.987147 20555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "1332353418394447a3543a8890e38e52"
format_stamp: "Formatted at 2026-09-23 18:05:38 on dist-test-slave-rvjj"
server_key: "9a8af5c284b825635fb99430781d6d05"
server_key_iv: "8037d576b99eb234a0ecdd9e689ea28c"
server_key_version: "encryptionkey@0"
I20260923 18:05:38.987385 20555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:39.001631 20555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:39.006364 20555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:39.006644 20555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:39.008739 20555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:39.008816 20555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:39.008880 20555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:39.008906 20555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:39.021991 20555 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:35883
I20260923 18:05:39.022472 20685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:35883 every 8 connection(s)
I20260923 18:05:39.024248 20555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:39.026996 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 20555
I20260923 18:05:39.027110 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:39.027165 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.010078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:48665 (local address 127.18.44.254:37413)
0923 18:05:39.010222 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.010226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.011962 (+ 1736us) server_negotiation.cc:407] Connection header received
0923 18:05:39.013324 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.013328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.013399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.013479 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.015163 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.015884 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.017328 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.017607 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.017706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.020704 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.020710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.020714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.020762 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.023433 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.024717 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.024722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.024724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.024886 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.025421 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.025425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.025427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.025749 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.025872 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.026676 (+ 804us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.026953 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:05:39.027463 18611 external_mini_cluster.cc:1428] Setting key b0a0dfe8ae920f497593be1a5237472f
I20260923 18:05:39.028445 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.010836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.194:48665)
0923 18:05:39.011681 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.011822 (+ 141us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.013039 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.013717 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.013736 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.014299 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:39.014987 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.015004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.016021 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.016024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.017182 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.017195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.017805 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.017810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.017988 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.018609 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:39.018628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:39.020408 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:39.023923 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.023931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.023949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.024547 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.025049 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.025053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.025056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.025315 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.025889 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:39.025899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:39.026230 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.026519 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.027020 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":168,"threads_started":1}
I20260923 18:05:39.030473 20686 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37413
I20260923 18:05:39.030794 20686 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:39.031709 20686 heartbeater.cc:511] Master 127.18.44.254:37413 requested a full tablet report, sending...
I20260923 18:05:39.033334 20343 ts_manager.cc:194] Registered new tserver with Master: 1332353418394447a3543a8890e38e52 (127.18.44.194:35883)
I20260923 18:05:39.034328 20343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:48665
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:39.042924 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.033381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.194:35079)
0923 18:05:39.033771 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.033785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.033914 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.034170 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.034173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.034419 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:39.034557 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.034564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.035883 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.035887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.036971 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.036987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.037528 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.037531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.037632 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.038342 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:39.038360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:39.038919 (+ 559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:39.040903 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.040907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.040909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.041152 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.041438 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.041441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.041443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.041551 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.042339 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:39.042342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:39.042465 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.042569 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.042725 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":127,"thread_start_us":145,"threads_started":1}
I20260923 18:05:39.043035 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.033455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:35079 (local address 127.18.44.254:37413)
0923 18:05:39.033609 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.033615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.033871 (+ 256us) server_negotiation.cc:407] Connection header received
0923 18:05:39.034009 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.034012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.034066 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.034136 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.034707 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.035620 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.037148 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.037393 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.037499 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.039148 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.039152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.039154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.039187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.040758 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.041260 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.041264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.041266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.041350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.041709 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.041715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.041717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.042190 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.042362 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.042660 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.042839 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:39.060456 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37413
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:39.188632 20695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:39.188943 20695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:39.189020 20695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:39.193921 20695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:39.194031 20695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:39.194092 20695 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:05:39.194113 20695 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:05:39.194169 20695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:39.200551 20695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:39.202006 20695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:39.203290 20695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:39.216118 20703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:39.216282 20700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:39.216638 20701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:39.218101 20695 server_base.cc:1034] running on GCE node
I20260923 18:05:39.219056 20695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:39.220083 20695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:39.221554 20695 hybrid_clock.cc:648] HybridClock initialized: now 1790186739221512 us; error 47 us; skew 500 ppm
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186739, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:39.226801 20695 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:39.228648 20695 webserver.cc:460] Webserver started at http://127.18.44.195:42501/ using document root <none> and password file <none>
I20260923 18:05:39.229522 20695 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:39.229624 20695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:39.230032 20695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:39.233357 20695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "f490e6b2302a47d79ba13bcde76bfd01"
format_stamp: "Formatted at 2026-09-23 18:05:39 on dist-test-slave-rvjj"
server_key: "d5872d3e360cd7d8b6564e316a28dabf"
server_key_iv: "1f0cfcdd5d3b5aad50e1ec496646c2cd"
server_key_version: "encryptionkey@0"
I20260923 18:05:39.234340 20695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "f490e6b2302a47d79ba13bcde76bfd01"
format_stamp: "Formatted at 2026-09-23 18:05:39 on dist-test-slave-rvjj"
server_key: "d5872d3e360cd7d8b6564e316a28dabf"
server_key_iv: "1f0cfcdd5d3b5aad50e1ec496646c2cd"
server_key_version: "encryptionkey@0"
I20260923 18:05:39.241828 20695 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 18:05:39.246424 20710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:39.248881 20695 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:05:39.249145 20695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "f490e6b2302a47d79ba13bcde76bfd01"
format_stamp: "Formatted at 2026-09-23 18:05:39 on dist-test-slave-rvjj"
server_key: "d5872d3e360cd7d8b6564e316a28dabf"
server_key_iv: "1f0cfcdd5d3b5aad50e1ec496646c2cd"
server_key_version: "encryptionkey@0"
I20260923 18:05:39.249336 20695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:39.279843 20695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:39.284866 20695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:39.285151 20695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:39.287240 20695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:39.287360 20695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:39.287425 20695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:39.287454 20695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186739, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:39.301018 20695 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:36529
I20260923 18:05:39.301134 20825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:36529 every 8 connection(s)
I20260923 18:05:39.303102 20695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:39.305632 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.288299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:47275 (local address 127.18.44.254:37413)
0923 18:05:39.288522 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.288529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.289478 (+ 949us) server_negotiation.cc:407] Connection header received
0923 18:05:39.290676 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.290693 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.290765 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.290823 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.292679 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.293448 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.295013 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.295392 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.295497 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.299819 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.299822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.299824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.299858 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.302323 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.303071 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.303075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.303077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.303188 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.303936 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.303941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.303943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.304395 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.304538 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.305314 (+ 776us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.305483 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:05:39.307125 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.288730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.195:47275)
0923 18:05:39.289288 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.289343 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.290405 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.291291 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.291301 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.291831 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:39.292517 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.292539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.293616 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.293619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.294848 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.294863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.295703 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.295708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.295939 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.296800 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:39.296835 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:39.299558 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:39.302518 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.302525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.302540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.302928 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.303604 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.303606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.303608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.303777 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.304570 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:39.304580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:39.304928 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.305179 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.305816 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":134,"threads_started":1}
I20260923 18:05:39.308959 20826 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37413
I20260923 18:05:39.309258 20826 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:39.310072 20826 heartbeater.cc:511] Master 127.18.44.254:37413 requested a full tablet report, sending...
I20260923 18:05:39.310527 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 20695
I20260923 18:05:39.310665 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:39.311043 18611 external_mini_cluster.cc:1428] Setting key ffad07141c26fdf29c7c641b4002f095
I20260923 18:05:39.311877 20343 ts_manager.cc:194] Registered new tserver with Master: f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:39.312716 20343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:47275
I20260923 18:05:39.313968 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:05:39.323048 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.311850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.18.44.195:60489)
0923 18:05:39.312371 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.312384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.312474 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.313645 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.313649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.313894 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:39.314048 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.314058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.315366 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.315371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.316562 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.316580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.317127 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.317131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.317236 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.317815 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:39.317833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:39.318312 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:39.320635 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.320641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.320644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.320989 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.321509 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.321512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.321515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.321726 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.322297 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:39.322302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:39.322476 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.322605 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.322829 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
I20260923 18:05:39.323426 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.313068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:60489 (local address 127.18.44.254:37413)
0923 18:05:39.313228 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.313234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.313288 (+ 54us) server_negotiation.cc:407] Connection header received
0923 18:05:39.313332 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.313335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.313390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.313487 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.314245 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.315044 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.316750 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.317000 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.317105 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.318545 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.318550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.318553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.318587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.320427 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.321259 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.321263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.321265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.321388 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.321872 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.321875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.321876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.322118 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.322265 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.322849 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.323036 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:39.347729 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.336191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.18.44.254:37413)
0923 18:05:39.336420 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.336425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.336522 (+ 97us) server_negotiation.cc:407] Connection header received
0923 18:05:39.336591 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.336594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.336651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.336761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.337682 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.338269 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.339490 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.339791 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.339939 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.343096 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.343100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.343102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.343140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.345036 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.345817 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.345821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.345822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.345953 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.346456 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.346459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.346461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.346780 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.346943 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.347330 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.347522 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 18:05:39.352568 20342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:39.358366 20342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:39.381865 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.372844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:36529 (local address 127.0.0.1:40224)
0923 18:05:39.374512 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.374586 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.375319 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.376881 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.376884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.376901 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.377019 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.377024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.378360 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.378363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.379770 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.379788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.381071 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.381074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.381206 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.381252 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.381391 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.381689 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":3,"thread_start_us":167,"threads_started":1}
I20260923 18:05:39.381980 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.373952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.18.44.195:36529)
0923 18:05:39.376415 (+ 2463us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.376424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.376447 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:05:39.376520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.376525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.376722 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.376889 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.377145 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.378232 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.379948 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.380754 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.380902 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.381163 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.381459 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.381814 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":180,"threads_started":1}
I20260923 18:05:39.383368 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.372844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35883 (local address 127.0.0.1:48866)
0923 18:05:39.374965 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.374994 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.375147 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.376127 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.376216 (+ 89us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.376262 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.376445 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.376454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.378304 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.378309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.380388 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.380404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.382719 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.382722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.382828 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.382853 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.383085 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.383186 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"mutex_wait_us":487,"thread_start_us":184,"threads_started":1}
I20260923 18:05:39.383517 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.373851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.18.44.194:35883)
0923 18:05:39.375550 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.375564 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.375596 (+ 32us) server_negotiation.cc:407] Connection header received
0923 18:05:39.375686 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.375692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.375929 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.376090 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.376606 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.377885 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.380593 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.381438 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.381813 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.382055 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.383162 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.383336 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1575,"thread_start_us":117,"threads_started":1}
I20260923 18:05:39.384375 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.374667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.18.44.193:33869)
0923 18:05:39.375099 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.375108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.376711 (+ 1603us) server_negotiation.cc:407] Connection header received
0923 18:05:39.376803 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.376807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.377079 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.377238 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.377418 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.378279 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.382165 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.383238 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.383433 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.383620 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.384083 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.384228 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":105,"threads_started":1}
I20260923 18:05:39.386665 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.374139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.0.0.1:48944)
0923 18:05:39.376485 (+ 2346us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.376502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.376589 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.377204 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.377207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.377223 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.377313 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.377319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.378415 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.378419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.380395 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.380404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.383730 (+ 3326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.383735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.383867 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.383899 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.384106 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.384267 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2270,"mutex_wait_us":359,"thread_start_us":132,"threads_started":1}
I20260923 18:05:39.388574 20760 tablet_service.cc:1467] Processing CreateTablet for tablet 69f1eb25bff648c68f2241e43e223637 (DEFAULT_TABLE table=test-table [id=a629d3a00dad422690ac86d5eaef08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.388573 20620 tablet_service.cc:1467] Processing CreateTablet for tablet 69f1eb25bff648c68f2241e43e223637 (DEFAULT_TABLE table=test-table [id=a629d3a00dad422690ac86d5eaef08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.390731 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f1eb25bff648c68f2241e43e223637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.390725 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f1eb25bff648c68f2241e43e223637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.390570 20480 tablet_service.cc:1467] Processing CreateTablet for tablet 69f1eb25bff648c68f2241e43e223637 (DEFAULT_TABLE table=test-table [id=a629d3a00dad422690ac86d5eaef08ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.392009 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f1eb25bff648c68f2241e43e223637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.400748 20849 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Bootstrap starting.
I20260923 18:05:39.403477 20849 tablet_bootstrap.cc:654] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.404471 20849 log.cc:824] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:39.405115 20850 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Bootstrap starting.
I20260923 18:05:39.405517 20851 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Bootstrap starting.
I20260923 18:05:39.407546 20849 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: No bootstrap required, opened a new log
I20260923 18:05:39.407830 20849 ts_tablet_manager.cc:1397] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:05:39.408532 20850 tablet_bootstrap.cc:654] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.409574 20850 log.cc:824] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:39.413621 20851 tablet_bootstrap.cc:654] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.413677 20849 raft_consensus.cc:348] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.414013 20849 raft_consensus.cc:374] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.414068 20849 raft_consensus.cc:729] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.414089 20850 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: No bootstrap required, opened a new log
I20260923 18:05:39.414283 20850 ts_tablet_manager.cc:1397] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260923 18:05:39.414629 20849 consensus_queue.cc:260] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.414700 20851 log.cc:824] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:39.415776 20546 heartbeater.cc:503] Master 127.18.44.254:37413 was elected leader, sending a full tablet report...
I20260923 18:05:39.415966 20849 ts_tablet_manager.cc:1428] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260923 18:05:39.417632 20851 tablet_bootstrap.cc:492] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: No bootstrap required, opened a new log
I20260923 18:05:39.418103 20851 ts_tablet_manager.cc:1397] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260923 18:05:39.420035 20850 raft_consensus.cc:348] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.420576 20850 raft_consensus.cc:374] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.420698 20850 raft_consensus.cc:729] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332353418394447a3543a8890e38e52, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.421406 20850 consensus_queue.cc:260] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.422674 20686 heartbeater.cc:503] Master 127.18.44.254:37413 was elected leader, sending a full tablet report...
I20260923 18:05:39.423213 20851 raft_consensus.cc:348] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.423635 20851 raft_consensus.cc:374] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.423666 20850 ts_tablet_manager.cc:1428] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260923 18:05:39.423724 20851 raft_consensus.cc:729] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.424853 20851 consensus_queue.cc:260] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.426164 20826 heartbeater.cc:503] Master 127.18.44.254:37413 was elected leader, sending a full tablet report...
I20260923 18:05:39.426553 20851 ts_tablet_manager.cc:1428] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260923 18:05:39.458566 20857 raft_consensus.cc:482] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:39.458932 20857 raft_consensus.cc:504] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.460678 20857 leader_election.cc:288] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1332353418394447a3543a8890e38e52 (127.18.44.194:35883), fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869)
W20260923 18:05:39.461550 20547 tablet.cc:2369] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:05:39.465309 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.460995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35883 (local address 127.18.44.195:54837)
0923 18:05:39.461139 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.461158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.461281 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.461763 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.461767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.461793 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.461918 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.461925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.462867 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.462871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.463926 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.463937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.464771 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.464775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.464906 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.464941 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.465039 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.465141 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 18:05:39.465363 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.461167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:54837 (local address 127.18.44.194:35883)
0923 18:05:39.461357 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.461363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.461383 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:05:39.461452 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.461455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.461528 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.461619 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.462052 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.462712 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.464124 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.464622 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.464735 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.464779 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.465112 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.465210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 18:05:39.466328 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f1eb25bff648c68f2241e43e223637" candidate_uuid: "f490e6b2302a47d79ba13bcde76bfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1332353418394447a3543a8890e38e52" is_pre_election: true
I20260923 18:05:39.466388 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.461167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.18.44.195:49575)
0923 18:05:39.461853 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.461870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.462003 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.462306 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.462309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.462327 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.462960 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.462969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.464075 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.464078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.464929 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.464937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.465900 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.465904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.465994 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.466021 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.466119 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.466214 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":174,"threads_started":1}
I20260923 18:05:39.466439 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.461500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:49575 (local address 127.18.44.193:33869)
0923 18:05:39.461694 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.461700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.461996 (+ 296us) server_negotiation.cc:407] Connection header received
0923 18:05:39.462124 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.462127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.462181 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.462263 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.463115 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.463921 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.465123 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.465717 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.465870 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.465924 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.466183 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.466314 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 18:05:39.466697 20640 raft_consensus.cc:2454] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f490e6b2302a47d79ba13bcde76bfd01 in term 0.
I20260923 18:05:39.467422 20711 leader_election.cc:302] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01; no voters:
I20260923 18:05:39.467556 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f1eb25bff648c68f2241e43e223637" candidate_uuid: "f490e6b2302a47d79ba13bcde76bfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" is_pre_election: true
I20260923 18:05:39.467921 20857 raft_consensus.cc:2781] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:39.467978 20500 raft_consensus.cc:2454] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f490e6b2302a47d79ba13bcde76bfd01 in term 0.
I20260923 18:05:39.468052 20857 raft_consensus.cc:482] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:39.468112 20857 raft_consensus.cc:3037] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.469429 20857 raft_consensus.cc:504] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.469846 20857 leader_election.cc:288] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [CANDIDATE]: Term 1 election: Requested vote from peers 1332353418394447a3543a8890e38e52 (127.18.44.194:35883), fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869)
I20260923 18:05:39.470700 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f1eb25bff648c68f2241e43e223637" candidate_uuid: "f490e6b2302a47d79ba13bcde76bfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1332353418394447a3543a8890e38e52"
I20260923 18:05:39.470479 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f1eb25bff648c68f2241e43e223637" candidate_uuid: "f490e6b2302a47d79ba13bcde76bfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
I20260923 18:05:39.470870 20640 raft_consensus.cc:3037] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.470914 20500 raft_consensus.cc:3037] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.473054 20500 raft_consensus.cc:2454] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f490e6b2302a47d79ba13bcde76bfd01 in term 1.
I20260923 18:05:39.473080 20640 raft_consensus.cc:2454] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f490e6b2302a47d79ba13bcde76bfd01 in term 1.
I20260923 18:05:39.473681 20711 leader_election.cc:302] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01; no voters:
I20260923 18:05:39.474033 20857 raft_consensus.cc:2781] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:39.474707 20857 raft_consensus.cc:686] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 LEADER]: Becoming Leader. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Running, Role: LEADER
I20260923 18:05:39.475133 20857 consensus_queue.cc:237] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } }
I20260923 18:05:39.480765 20343 catalog_manager.cc:5495] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 reported cstate change: term changed from 0 to 1, leader changed from <none> to f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "f490e6b2302a47d79ba13bcde76bfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } health_report { overall_health: UNKNOWN } } }
W20260923 18:05:39.526068 20687 tablet.cc:2369] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 18:05:39 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186738, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:39.554167 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.537973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37413 (local address 127.0.0.1:40220)
0923 18:05:39.538545 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.538576 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.538957 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.539813 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.539816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.540370 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:39.540506 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.540513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.541607 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.541611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.542260 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.542272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.542929 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.542933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.543018 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.543960 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:39.543985 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:39.546547 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:39.550739 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.550756 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.550763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.551158 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.552141 (+ 983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:39.552145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:39.552147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:39.552340 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:39.553412 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:39.553425 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:39.553608 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.553801 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.554003 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":194,"threads_started":1}
W20260923 18:05:39.554718 20827 tablet.cc:2369] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:05:39.555226 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.538308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.18.44.254:37413)
0923 18:05:39.538484 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.538489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.538861 (+ 372us) server_negotiation.cc:407] Connection header received
0923 18:05:39.539351 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.539354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.539420 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.539640 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:39.540703 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.541351 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.542469 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.542738 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.542870 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.547174 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:39.547179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:39.547193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:39.547241 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:39.550370 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.551764 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.551768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.551770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.551945 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:39.552572 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:39.552576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:39.552578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:39.552894 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:39.553049 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.554028 (+ 979us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.554207 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:05:39.561926 20343 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40220:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:05:39.562887 20343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:39.572793 20620 tablet_service.cc:1467] Processing CreateTablet for tablet b3541cd8d7bc4de4975c427387fa979c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f521a5a1944028882dda6b41ba045d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:39.573601 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3541cd8d7bc4de4975c427387fa979c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.573132 20480 tablet_service.cc:1467] Processing CreateTablet for tablet b3541cd8d7bc4de4975c427387fa979c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f521a5a1944028882dda6b41ba045d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:39.573000 20760 tablet_service.cc:1467] Processing CreateTablet for tablet b3541cd8d7bc4de4975c427387fa979c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f521a5a1944028882dda6b41ba045d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:05:39.574220 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3541cd8d7bc4de4975c427387fa979c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.574220 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3541cd8d7bc4de4975c427387fa979c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.577039 20850 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: Bootstrap starting.
I20260923 18:05:39.578706 20850 tablet_bootstrap.cc:654] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.579246 20849 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: Bootstrap starting.
I20260923 18:05:39.579757 20851 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: Bootstrap starting.
I20260923 18:05:39.580364 20850 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: No bootstrap required, opened a new log
I20260923 18:05:39.580466 20850 ts_tablet_manager.cc:1397] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:39.580904 20849 tablet_bootstrap.cc:654] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.581216 20850 raft_consensus.cc:348] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.581362 20850 raft_consensus.cc:374] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.581403 20850 raft_consensus.cc:729] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332353418394447a3543a8890e38e52, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.581540 20850 consensus_queue.cc:260] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.581740 20851 tablet_bootstrap.cc:654] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.581817 20850 ts_tablet_manager.cc:1428] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:05:39.582239 20856 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.582442 20856 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.583745 20849 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: No bootstrap required, opened a new log
I20260923 18:05:39.583967 20851 tablet_bootstrap.cc:492] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: No bootstrap required, opened a new log
I20260923 18:05:39.584018 20849 ts_tablet_manager.cc:1397] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:39.584102 20851 ts_tablet_manager.cc:1397] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260923 18:05:39.584844 20849 raft_consensus.cc:348] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.584733 20851 raft_consensus.cc:348] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.585001 20849 raft_consensus.cc:374] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.585001 20851 raft_consensus.cc:374] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.585035 20849 raft_consensus.cc:729] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.585035 20851 raft_consensus.cc:729] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.585196 20849 consensus_queue.cc:260] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.585180 20851 consensus_queue.cc:260] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.585644 20849 ts_tablet_manager.cc:1428] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:05:39.585790 20851 ts_tablet_manager.cc:1428] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 18:05:39.585762 20854 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.586050 20854 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.586107 20857 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.586280 20857 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.686808 20854 raft_consensus.cc:482] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:39.687077 20854 raft_consensus.cc:504] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.689332 20854 leader_election.cc:288] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1332353418394447a3543a8890e38e52 (127.18.44.194:35883), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:39.694890 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.690108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:45351 (local address 127.18.44.194:35883)
0923 18:05:39.690288 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.690293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.690591 (+ 298us) server_negotiation.cc:407] Connection header received
0923 18:05:39.690808 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.690813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.690874 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.690975 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.691413 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.691968 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.693804 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.694253 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.694435 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.694497 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.694670 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.694749 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 18:05:39.694958 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.689694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35883 (local address 127.18.44.193:45351)
0923 18:05:39.690469 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.690482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.690625 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.691009 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.691012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.691044 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.691208 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.691215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.692096 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.692100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.693650 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.693665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.694433 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.694438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.694535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.694561 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.694637 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.694732 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":148,"threads_started":1}
I20260923 18:05:39.695911 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3541cd8d7bc4de4975c427387fa979c" candidate_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1332353418394447a3543a8890e38e52" is_pre_election: true
I20260923 18:05:39.696085 20640 raft_consensus.cc:2454] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee495ca8b1a4908b71d4e1ed8024aac in term 0.
I20260923 18:05:39.696502 20431 leader_election.cc:302] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, fee495ca8b1a4908b71d4e1ed8024aac; no voters:
I20260923 18:05:39.696975 20854 raft_consensus.cc:2781] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:39.697158 20854 raft_consensus.cc:482] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:39.697220 20854 raft_consensus.cc:3037] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.697329 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.690065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:55951 (local address 127.18.44.195:36529)
0923 18:05:39.690263 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.690268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.690692 (+ 424us) server_negotiation.cc:407] Connection header received
0923 18:05:39.690969 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.690972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.691031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.691138 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:39.692230 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.693003 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.695449 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.696377 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.696612 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.696685 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.696958 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.697181 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 18:05:39.697655 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.689694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:36529 (local address 127.18.44.193:55951)
0923 18:05:39.690540 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.690561 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.690738 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.691198 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.691201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.691223 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:39.692048 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.692061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.693173 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.693178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.694707 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.694829 (+ 122us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.696586 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.696590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.696793 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.696825 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.696914 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.697023 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":396,"thread_start_us":206,"threads_started":1}
I20260923 18:05:39.698392 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3541cd8d7bc4de4975c427387fa979c" candidate_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01" is_pre_election: true
I20260923 18:05:39.698649 20764 raft_consensus.cc:2454] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee495ca8b1a4908b71d4e1ed8024aac in term 0.
I20260923 18:05:39.699031 20854 raft_consensus.cc:504] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.699993 20854 leader_election.cc:288] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [CANDIDATE]: Term 1 election: Requested vote from peers 1332353418394447a3543a8890e38e52 (127.18.44.194:35883), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:39.700358 20640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3541cd8d7bc4de4975c427387fa979c" candidate_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1332353418394447a3543a8890e38e52"
I20260923 18:05:39.700397 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3541cd8d7bc4de4975c427387fa979c" candidate_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01"
I20260923 18:05:39.700520 20764 raft_consensus.cc:3037] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.700528 20640 raft_consensus.cc:3037] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:39.701937 20640 raft_consensus.cc:2454] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee495ca8b1a4908b71d4e1ed8024aac in term 1.
I20260923 18:05:39.701937 20764 raft_consensus.cc:2454] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee495ca8b1a4908b71d4e1ed8024aac in term 1.
I20260923 18:05:39.702347 20433 leader_election.cc:302] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f490e6b2302a47d79ba13bcde76bfd01, fee495ca8b1a4908b71d4e1ed8024aac; no voters:
I20260923 18:05:39.702658 20854 raft_consensus.cc:2781] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:39.703239 20854 raft_consensus.cc:686] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [term 1 LEADER]: Becoming Leader. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Running, Role: LEADER
I20260923 18:05:39.704200 20854 consensus_queue.cc:237] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.707871 20872 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: TxnStatusTablet state changed. Reason: New leader fee495ca8b1a4908b71d4e1ed8024aac. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.709260 20872 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:39.709748 20874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:05:39.710052 20874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:05:39.710080 20342 catalog_manager.cc:5495] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac reported cstate change: term changed from 0 to 1, leader changed from <none> to fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193). New cstate: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:39.752696 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.745194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.18.44.193:33869)
0923 18:05:39.745328 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.745333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.745424 (+ 91us) server_negotiation.cc:407] Connection header received
0923 18:05:39.745682 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.745686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.745757 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.745861 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:39.746376 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.747403 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.749707 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.750138 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.750273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.751374 (+ 1101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.752077 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.752249 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.752422 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.752506 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:05:39.752796 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.745039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.0.0.1:48952)
0923 18:05:39.745301 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.745322 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.745509 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.745965 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.745971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.745990 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:39.746187 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.746196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.747810 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.747815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.749408 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.749440 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.750705 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.750711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.750872 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.751013 (+ 141us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.752252 (+ 1239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.752278 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.752386 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.752570 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 18:05:39.763242 20764 raft_consensus.cc:1261] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Refusing update from remote peer fee495ca8b1a4908b71d4e1ed8024aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:39.763242 20640 raft_consensus.cc:1261] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Refusing update from remote peer fee495ca8b1a4908b71d4e1ed8024aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:39.764206 20872 consensus_queue.cc:1035] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:39.764496 20854 consensus_queue.cc:1035] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:39.769265 20857 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: TxnStatusTablet state changed. Reason: New leader fee495ca8b1a4908b71d4e1ed8024aac. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.769503 20857 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.769809 20856 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: TxnStatusTablet state changed. Reason: New leader fee495ca8b1a4908b71d4e1ed8024aac. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.770011 20856 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.774466 20872 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.774451 20857 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.774613 20872 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:39.774613 20857 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P f490e6b2302a47d79ba13bcde76bfd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.777208 20872 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.777382 20872 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P fee495ca8b1a4908b71d4e1ed8024aac: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:05:39.781884 20856 tablet_replica.cc:439] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } } }
I20260923 18:05:39.782069 20856 tablet_replica.cc:442] T b3541cd8d7bc4de4975c427387fa979c P 1332353418394447a3543a8890e38e52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:05:39.783098 20878 mvcc.cc:205] Tried to move back new op lower bound from 7332604886058176512 to 7332604885833342976. Current Snapshot: MvccSnapshot[applied={T|T < 7332604886058176512}]
I20260923 18:05:39.784569 20879 mvcc.cc:205] Tried to move back new op lower bound from 7332604886058176512 to 7332604885833342976. Current Snapshot: MvccSnapshot[applied={T|T < 7332604886058176512}]
I20260923 18:05:39.794097 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.788846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.18.44.195:36529)
0923 18:05:39.788992 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.788996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.789068 (+ 72us) server_negotiation.cc:407] Connection header received
0923 18:05:39.789267 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.789271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.789339 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.789459 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:39.789700 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.790259 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.792058 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.792416 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.792529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.792979 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.793619 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.793759 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.793867 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.793941 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:05:39.818604 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.813729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:55239 (local address 127.18.44.193:33869)
0923 18:05:39.813858 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.813861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.814117 (+ 256us) server_negotiation.cc:407] Connection header received
0923 18:05:39.814294 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.814296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.814348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.814439 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:39.814829 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.815469 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.816884 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.817139 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.817262 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.818021 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.818115 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.818185 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.818386 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.818459 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:05:39.818681 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.813632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.18.44.195:55239)
0923 18:05:39.814016 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.814034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.814174 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.814477 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.814480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.814488 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:39.814665 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.814673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.815591 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.815595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.816697 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.816710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.817301 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.817305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.817421 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.817513 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.818216 (+ 703us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.818225 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.818311 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.818447 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260923 18:05:39.829061 20640 raft_consensus.cc:1261] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Refusing update from remote peer f490e6b2302a47d79ba13bcde76bfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:39.829641 20859 consensus_queue.cc:1035] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:39.830057 20500 raft_consensus.cc:1261] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Refusing update from remote peer f490e6b2302a47d79ba13bcde76bfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:39.830689 20859 consensus_queue.cc:1035] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:39.837297 20877 mvcc.cc:205] Tried to move back new op lower bound from 7332604886333825024 to 7332604884896137216. Current Snapshot: MvccSnapshot[applied={T|T < 7332604886333825024}]
I20260923 18:05:39.867807 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.862870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38613 (local address 127.18.44.195:36529)
0923 18:05:39.862998 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:39.863003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:39.863390 (+ 387us) server_negotiation.cc:407] Connection header received
0923 18:05:39.863595 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:39.863599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:39.863655 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:39.863725 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:39.864483 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.865236 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.866259 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.866506 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.866600 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.867128 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.867221 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.867356 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:39.867561 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 18:05:39.867629 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:05:39.867933 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:39.862759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:36529 (local address 127.18.44.193:38613)
0923 18:05:39.863290 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:39.863305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:39.863491 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:39.864053 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:39.864055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:39.864062 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:39.864302 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.864314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.865400 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.865404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.866132 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:39.866142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:39.866797 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:39.866800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:39.866904 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:39.866974 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:39.867382 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:39.867397 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 18:05:39.867536 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 18:05:39.867689 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":139,"threads_started":1}
I20260923 18:05:39.912467 20343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:39.913285 20343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:39.918800 20343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Sending DeleteTablet for 3 replicas of tablet 69f1eb25bff648c68f2241e43e223637
I20260923 18:05:39.920186 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f1eb25bff648c68f2241e43e223637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:39 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48866
I20260923 18:05:39.921275 20896 tablet_replica.cc:330] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: stopping tablet replica
I20260923 18:05:39.921864 20896 raft_consensus.cc:2229] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:39.922529 20896 raft_consensus.cc:2258] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:39.921253 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f1eb25bff648c68f2241e43e223637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:39 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40224
I20260923 18:05:39.925562 20896 ts_tablet_manager.cc:1905] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:39.928869 20343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:05:39.930546 20480 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f1eb25bff648c68f2241e43e223637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:39 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48944
I20260923 18:05:39.931057 20896 ts_tablet_manager.cc:1918] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:39.931245 20896 log.cc:1196] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/69f1eb25bff648c68f2241e43e223637
W20260923 18:05:39.931479 20343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:39.931914 20896 ts_tablet_manager.cc:1939] T 69f1eb25bff648c68f2241e43e223637 P 1332353418394447a3543a8890e38e52: Deleting consensus metadata
I20260923 18:05:39.933624 20327 catalog_manager.cc:4841] TS 1332353418394447a3543a8890e38e52 (127.18.44.194:35883): tablet 69f1eb25bff648c68f2241e43e223637 (table test-table [id=a629d3a00dad422690ac86d5eaef08ad]) successfully deleted
I20260923 18:05:39.934252 20898 tablet_replica.cc:330] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: stopping tablet replica
I20260923 18:05:39.934705 20898 raft_consensus.cc:2229] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:39.934976 20898 raft_consensus.cc:2258] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:39.935448 20897 tablet_replica.cc:330] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: stopping tablet replica
I20260923 18:05:39.935865 20897 raft_consensus.cc:2229] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:39.936327 20897 raft_consensus.cc:2258] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:39.936901 20898 ts_tablet_manager.cc:1905] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:39.941946 20898 ts_tablet_manager.cc:1918] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:39.942148 20898 log.cc:1196] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/69f1eb25bff648c68f2241e43e223637
I20260923 18:05:39.942272 20620 tablet_service.cc:1467] Processing CreateTablet for tablet 17e185261842463082cf54df5f6fd86d (DEFAULT_TABLE table=test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.942662 20898 ts_tablet_manager.cc:1939] T 69f1eb25bff648c68f2241e43e223637 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting consensus metadata
I20260923 18:05:39.942844 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e185261842463082cf54df5f6fd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.943369 20480 tablet_service.cc:1467] Processing CreateTablet for tablet 17e185261842463082cf54df5f6fd86d (DEFAULT_TABLE table=test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.943833 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e185261842463082cf54df5f6fd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.945781 20330 catalog_manager.cc:4841] TS fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869): tablet 69f1eb25bff648c68f2241e43e223637 (table test-table [id=a629d3a00dad422690ac86d5eaef08ad]) successfully deleted
I20260923 18:05:39.948033 20850 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Bootstrap starting.
I20260923 18:05:39.948176 20760 tablet_service.cc:1467] Processing CreateTablet for tablet 17e185261842463082cf54df5f6fd86d (DEFAULT_TABLE table=test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:39.948698 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e185261842463082cf54df5f6fd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:39.951328 20850 tablet_bootstrap.cc:654] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.952150 20849 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Bootstrap starting.
I20260923 18:05:39.953224 20849 tablet_bootstrap.cc:654] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.954581 20851 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Bootstrap starting.
I20260923 18:05:39.956825 20851 tablet_bootstrap.cc:654] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:39.960096 20897 ts_tablet_manager.cc:1905] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:39.960498 20851 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: No bootstrap required, opened a new log
I20260923 18:05:39.962131 20851 ts_tablet_manager.cc:1397] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260923 18:05:39.963883 20851 raft_consensus.cc:348] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.965806 20849 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: No bootstrap required, opened a new log
I20260923 18:05:39.965948 20849 ts_tablet_manager.cc:1397] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260923 18:05:39.965682 20851 raft_consensus.cc:374] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.966143 20851 raft_consensus.cc:729] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.966463 20849 raft_consensus.cc:348] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.966610 20849 raft_consensus.cc:374] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.966646 20849 raft_consensus.cc:729] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.966797 20849 consensus_queue.cc:260] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.966578 20851 consensus_queue.cc:260] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.967077 20849 ts_tablet_manager.cc:1428] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 18:05:39.964856 20897 ts_tablet_manager.cc:1918] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:39.967378 20897 log.cc:1196] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/69f1eb25bff648c68f2241e43e223637
I20260923 18:05:39.967823 20897 ts_tablet_manager.cc:1939] T 69f1eb25bff648c68f2241e43e223637 P f490e6b2302a47d79ba13bcde76bfd01: Deleting consensus metadata
I20260923 18:05:39.967859 20850 tablet_bootstrap.cc:492] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: No bootstrap required, opened a new log
I20260923 18:05:39.967974 20850 ts_tablet_manager.cc:1397] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20260923 18:05:39.968612 20850 raft_consensus.cc:348] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.968775 20850 raft_consensus.cc:374] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:39.968820 20850 raft_consensus.cc:729] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332353418394447a3543a8890e38e52, State: Initialized, Role: FOLLOWER
I20260923 18:05:39.969540 20850 consensus_queue.cc:260] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:39.969619 20329 catalog_manager.cc:4841] TS f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529): tablet 69f1eb25bff648c68f2241e43e223637 (table test-table [id=a629d3a00dad422690ac86d5eaef08ad]) successfully deleted
I20260923 18:05:39.969954 20850 ts_tablet_manager.cc:1428] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 18:05:39.970657 20851 ts_tablet_manager.cc:1428] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260923 18:05:40.078086 20856 raft_consensus.cc:482] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:40.078293 20856 raft_consensus.cc:504] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:40.080044 20856 leader_election.cc:288] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:40.086165 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.080500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.18.44.194:33511)
0923 18:05:40.081438 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:40.081456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:40.081691 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:40.082291 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:40.082295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:40.082319 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:40.082446 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.082453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.083537 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.083542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.084473 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.084490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.085440 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.085445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.085643 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.085675 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 18:05:40.085781 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 18:05:40.085885 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":446,"thread_start_us":153,"threads_started":1}
I20260923 18:05:40.086172 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.080650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:33511 (local address 127.18.44.193:33869)
0923 18:05:40.080836 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:40.080841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:40.081590 (+ 749us) server_negotiation.cc:407] Connection header received
0923 18:05:40.081924 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:40.081929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:40.082002 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:40.082143 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:40.082610 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.083274 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.084655 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.085254 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.085418 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.085513 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:40.085836 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 18:05:40.085963 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 18:05:40.086192 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.080756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:33813 (local address 127.18.44.195:36529)
0923 18:05:40.080904 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:40.080909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:40.081456 (+ 547us) server_negotiation.cc:407] Connection header received
0923 18:05:40.081646 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:40.081650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:40.081706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:40.081788 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:40.082099 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.082774 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.084862 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.085463 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.085619 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.085674 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:40.085963 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 18:05:40.086055 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:05:40.086772 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.080500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:36529 (local address 127.18.44.194:33813)
0923 18:05:40.081180 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:40.081198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:40.081487 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:40.081834 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:40.081836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:40.081851 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:40.081985 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.081991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.082926 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.082930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.084660 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.084679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.085668 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.085673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.085801 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.085832 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 18:05:40.086425 (+ 593us) client_negotiation.cc:240] Negotiation successful
0923 18:05:40.086572 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":1,"thread_start_us":168,"threads_started":1}
I20260923 18:05:40.087492 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e185261842463082cf54df5f6fd86d" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" is_pre_election: true
I20260923 18:05:40.087744 20500 raft_consensus.cc:2454] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:40.088042 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e185261842463082cf54df5f6fd86d" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01" is_pre_election: true
I20260923 18:05:40.088178 20764 raft_consensus.cc:2454] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:40.088434 20574 leader_election.cc:302] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, fee495ca8b1a4908b71d4e1ed8024aac; no voters:
I20260923 18:05:40.089022 20856 raft_consensus.cc:2781] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:40.089113 20856 raft_consensus.cc:482] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:40.089151 20856 raft_consensus.cc:3037] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:40.091161 20856 raft_consensus.cc:504] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:40.091912 20856 leader_election.cc:288] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Requested vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:40.092453 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e185261842463082cf54df5f6fd86d" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01"
I20260923 18:05:40.092621 20764 raft_consensus.cc:3037] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:40.093932 20764 raft_consensus.cc:2454] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:40.094337 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e185261842463082cf54df5f6fd86d" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
I20260923 18:05:40.094491 20500 raft_consensus.cc:3037] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:40.094707 20573 leader_election.cc:302] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01; no voters:
I20260923 18:05:40.095445 20856 raft_consensus.cc:2781] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:40.095711 20856 raft_consensus.cc:686] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Becoming Leader. State: Replica: 1332353418394447a3543a8890e38e52, State: Running, Role: LEADER
I20260923 18:05:40.095885 20500 raft_consensus.cc:2454] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:40.096113 20856 consensus_queue.cc:237] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:40.100055 20343 catalog_manager.cc:5495] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1332353418394447a3543a8890e38e52 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "1332353418394447a3543a8890e38e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:40.146071 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.141286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.18.44.194:35883)
0923 18:05:40.141489 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:40.141493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:40.141513 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:05:40.141643 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:40.141646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:40.141713 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:40.141833 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:40.142396 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.142932 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.144326 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.144577 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.144701 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.145082 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:40.145591 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:40.145703 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:40.145863 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 18:05:40.145939 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 18:05:40.164258 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.159031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.18.44.194:36591)
0923 18:05:40.159531 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:40.159552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:40.159683 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:40.160245 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:40.160248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:40.160256 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:40.160530 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.160541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.161470 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.161473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.162142 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.162153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.162966 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.162969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.163073 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.163139 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:40.163884 (+ 745us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:40.163892 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:05:40.163983 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 18:05:40.164072 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":280,"threads_started":1}
I20260923 18:05:40.164251 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.159391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:36591 (local address 127.18.44.193:33869)
0923 18:05:40.159555 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:40.159561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:40.159631 (+ 70us) server_negotiation.cc:407] Connection header received
0923 18:05:40.159991 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:40.159994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:40.160070 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:40.160185 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:40.160692 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.161280 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.162339 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.162765 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.162907 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.163454 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:40.163737 (+ 283us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:40.163832 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:40.164006 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 18:05:40.164087 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:05:40.172504 20500 raft_consensus.cc:1261] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:40.172863 20764 raft_consensus.cc:1261] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:40.173259 20856 consensus_queue.cc:1035] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:40.173691 20904 consensus_queue.cc:1035] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:40.199291 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.194999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35883 (local address 127.18.44.193:39949)
0923 18:05:40.195242 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:40.195292 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:40.195401 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:40.195886 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:40.195888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:40.195895 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:05:40.196000 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.196006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.196917 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.196919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.197617 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:40.197626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.198245 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.198248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:40.198310 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.198340 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:05:40.198941 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:05:40.198949 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:05:40.199022 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:05:40.199112 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 18:05:40.199525 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:40.195082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:39949 (local address 127.18.44.194:35883)
0923 18:05:40.195240 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:40.195245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:40.195340 (+ 95us) server_negotiation.cc:407] Connection header received
0923 18:05:40.195669 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:40.195671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:40.195788 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:40.195870 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:40.196113 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.196646 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.197738 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:40.198023 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:40.198150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:40.198524 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:40.198681 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:40.198764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:40.199105 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 18:05:40.199190 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 18:05:40.200591 20878 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 18:05:41.800616 20343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:41.800833 20343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:41.804869 20343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Sending DeleteTablet for 3 replicas of tablet 17e185261842463082cf54df5f6fd86d
I20260923 18:05:41.805616 20480 tablet_service.cc:1514] Processing DeleteTablet for tablet 17e185261842463082cf54df5f6fd86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:41 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48944
I20260923 18:05:41.805720 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet 17e185261842463082cf54df5f6fd86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:41 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40224
I20260923 18:05:41.805905 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet 17e185261842463082cf54df5f6fd86d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:41 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48866
I20260923 18:05:41.806082 20935 tablet_replica.cc:330] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: stopping tablet replica
I20260923 18:05:41.806239 20935 raft_consensus.cc:2229] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:41.806298 20936 tablet_replica.cc:330] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: stopping tablet replica
I20260923 18:05:41.806391 20935 raft_consensus.cc:2258] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:41.806403 20936 raft_consensus.cc:2229] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:41.806522 20936 raft_consensus.cc:2258] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:41.807195 20935 ts_tablet_manager.cc:1905] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:41.807652 20936 ts_tablet_manager.cc:1905] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:41.811496 20937 tablet_replica.cc:330] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: stopping tablet replica
I20260923 18:05:41.811815 20937 raft_consensus.cc:2229] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:41.811873 20935 ts_tablet_manager.cc:1918] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:05:41.811873 20936 ts_tablet_manager.cc:1918] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:05:41.811975 20936 log.cc:1196] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/17e185261842463082cf54df5f6fd86d
I20260923 18:05:41.812054 20935 log.cc:1196] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/17e185261842463082cf54df5f6fd86d
I20260923 18:05:41.812228 20936 ts_tablet_manager.cc:1939] T 17e185261842463082cf54df5f6fd86d P f490e6b2302a47d79ba13bcde76bfd01: Deleting consensus metadata
I20260923 18:05:41.812312 20935 ts_tablet_manager.cc:1939] T 17e185261842463082cf54df5f6fd86d P fee495ca8b1a4908b71d4e1ed8024aac: Deleting consensus metadata
I20260923 18:05:41.812597 20937 raft_consensus.cc:2258] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:41.813205 20330 catalog_manager.cc:4841] TS fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869): tablet 17e185261842463082cf54df5f6fd86d (table test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]) successfully deleted
I20260923 18:05:41.813205 20329 catalog_manager.cc:4841] TS f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529): tablet 17e185261842463082cf54df5f6fd86d (table test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]) successfully deleted
I20260923 18:05:41.813587 20937 ts_tablet_manager.cc:1905] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:41.816937 20937 ts_tablet_manager.cc:1918] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:05:41.817045 20937 log.cc:1196] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/17e185261842463082cf54df5f6fd86d
I20260923 18:05:41.817317 20937 ts_tablet_manager.cc:1939] T 17e185261842463082cf54df5f6fd86d P 1332353418394447a3543a8890e38e52: Deleting consensus metadata
I20260923 18:05:41.818312 20327 catalog_manager.cc:4841] TS 1332353418394447a3543a8890e38e52 (127.18.44.194:35883): tablet 17e185261842463082cf54df5f6fd86d (table test-table [id=d8dd7831897f4d61a669fbfd7265ffa5]) successfully deleted
Sep 23 18:05:41 dist-test-slave-rvjj krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:05:41 dist-test-slave-rvjj krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:41.846634 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:41.834274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.18.44.254:37413)
0923 18:05:41.834603 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:41.834611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:41.834981 (+ 370us) server_negotiation.cc:407] Connection header received
0923 18:05:41.835166 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:41.835169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:41.835230 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:41.835411 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:41.836470 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:41.837065 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:41.838203 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:41.838482 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:41.838603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:41.842273 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:41.842277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:41.842280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:41.842319 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:41.844311 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:41.844957 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:41.844961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:41.844962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:41.845057 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:41.845525 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:41.845528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:41.845529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:41.845842 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:41.845969 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:41.846349 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 18:05:41.846488 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
I20260923 18:05:41.850594 20343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:41.851086 20343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:41.856889 20480 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (DEFAULT_TABLE table=test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:41.856889 20760 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (DEFAULT_TABLE table=test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:41.857223 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:41.857223 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:41.857465 20620 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (DEFAULT_TABLE table=test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:41.857752 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:41.860357 20946 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Bootstrap starting.
I20260923 18:05:41.860842 20947 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Bootstrap starting.
I20260923 18:05:41.861327 20946 tablet_bootstrap.cc:654] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:41.861538 20948 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Bootstrap starting.
I20260923 18:05:41.862337 20947 tablet_bootstrap.cc:654] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:41.862490 20948 tablet_bootstrap.cc:654] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:41.862970 20946 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: No bootstrap required, opened a new log
I20260923 18:05:41.863178 20946 ts_tablet_manager.cc:1397] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:41.863897 20946 raft_consensus.cc:348] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.864105 20946 raft_consensus.cc:374] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:41.864125 20947 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: No bootstrap required, opened a new log
I20260923 18:05:41.864148 20946 raft_consensus.cc:729] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Initialized, Role: FOLLOWER
I20260923 18:05:41.864250 20947 ts_tablet_manager.cc:1397] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:41.864336 20946 consensus_queue.cc:260] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.864796 20948 tablet_bootstrap.cc:492] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: No bootstrap required, opened a new log
I20260923 18:05:41.864773 20947 raft_consensus.cc:348] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.864908 20948 ts_tablet_manager.cc:1397] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:41.864910 20947 raft_consensus.cc:374] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:41.864923 20946 ts_tablet_manager.cc:1428] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 18:05:41.864948 20947 raft_consensus.cc:729] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Initialized, Role: FOLLOWER
I20260923 18:05:41.865075 20947 consensus_queue.cc:260] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.865370 20947 ts_tablet_manager.cc:1428] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:05:41.865460 20948 raft_consensus.cc:348] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.865640 20948 raft_consensus.cc:374] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:41.865695 20948 raft_consensus.cc:729] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332353418394447a3543a8890e38e52, State: Initialized, Role: FOLLOWER
I20260923 18:05:41.865859 20948 consensus_queue.cc:260] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.866281 20948 ts_tablet_manager.cc:1428] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 18:05:41.919518 20926 raft_consensus.cc:482] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:41.919741 20926 raft_consensus.cc:504] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.920269 20926 leader_election.cc:288] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:41.920887 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc9c7c611e74da4b20ac7bf9ae624a7" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" is_pre_election: true
I20260923 18:05:41.920938 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc9c7c611e74da4b20ac7bf9ae624a7" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01" is_pre_election: true
I20260923 18:05:41.921094 20500 raft_consensus.cc:2454] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:41.921128 20765 raft_consensus.cc:2454] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:41.921556 20573 leader_election.cc:302] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01; no voters:
I20260923 18:05:41.921839 20926 raft_consensus.cc:2781] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:41.921950 20926 raft_consensus.cc:482] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:41.922034 20926 raft_consensus.cc:3037] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:41.923458 20926 raft_consensus.cc:504] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.923971 20926 leader_election.cc:288] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Requested vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:41.924494 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc9c7c611e74da4b20ac7bf9ae624a7" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01"
I20260923 18:05:41.924543 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc9c7c611e74da4b20ac7bf9ae624a7" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
I20260923 18:05:41.924625 20765 raft_consensus.cc:3037] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:41.924726 20500 raft_consensus.cc:3037] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:41.925644 20765 raft_consensus.cc:2454] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:41.925949 20500 raft_consensus.cc:2454] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:41.926059 20573 leader_election.cc:302] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01; no voters:
I20260923 18:05:41.926275 20926 raft_consensus.cc:2781] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:41.926373 20926 raft_consensus.cc:686] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Becoming Leader. State: Replica: 1332353418394447a3543a8890e38e52, State: Running, Role: LEADER
I20260923 18:05:41.926558 20926 consensus_queue.cc:237] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:41.928690 20341 catalog_manager.cc:5495] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1332353418394447a3543a8890e38e52 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "1332353418394447a3543a8890e38e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:41.961881 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:41.958569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.18.44.194:35883)
0923 18:05:41.958910 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:41.958918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:41.958931 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:05:41.958973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:41.958975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:41.959017 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:41.959096 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:41.959401 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:41.959945 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:41.960779 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:41.961030 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:41.961114 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:41.961383 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:41.961462 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:41.961522 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:41.961653 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 18:05:41.961725 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":145,"threads_started":1}
I20260923 18:05:41.970090 20765 raft_consensus.cc:1261] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:41.970140 20500 raft_consensus.cc:1261] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:41.970603 20926 consensus_queue.cc:1035] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:41.970816 20929 consensus_queue.cc:1035] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:42.014127 20341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:42.014312 20341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:42.017299 20341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Sending DeleteTablet for 3 replicas of tablet 5cc9c7c611e74da4b20ac7bf9ae624a7
I20260923 18:05:42.018214 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:42 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40224
I20260923 18:05:42.018214 20480 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:42 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48944
I20260923 18:05:42.018452 20935 tablet_replica.cc:330] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: stopping tablet replica
I20260923 18:05:42.018452 20936 tablet_replica.cc:330] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: stopping tablet replica
I20260923 18:05:42.018404 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:42 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48866
I20260923 18:05:42.018581 20935 raft_consensus.cc:2229] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:42.018581 20936 raft_consensus.cc:2229] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:42.018709 20936 raft_consensus.cc:2258] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:42.018708 20935 raft_consensus.cc:2258] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:42.018805 20937 tablet_replica.cc:330] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: stopping tablet replica
I20260923 18:05:42.018958 20937 raft_consensus.cc:2229] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:42.019232 20937 raft_consensus.cc:2258] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:42.019886 20936 ts_tablet_manager.cc:1905] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:42.020169 20937 ts_tablet_manager.cc:1905] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:42.020169 20935 ts_tablet_manager.cc:1905] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:42.020128 20341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:42.020725 20341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:42.023195 20937 ts_tablet_manager.cc:1918] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:42.023195 20935 ts_tablet_manager.cc:1918] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:42.023322 20935 log.cc:1196] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/5cc9c7c611e74da4b20ac7bf9ae624a7
I20260923 18:05:42.023322 20937 log.cc:1196] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/5cc9c7c611e74da4b20ac7bf9ae624a7
I20260923 18:05:42.023597 20935 ts_tablet_manager.cc:1939] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P fee495ca8b1a4908b71d4e1ed8024aac: Deleting consensus metadata
I20260923 18:05:42.023598 20937 ts_tablet_manager.cc:1939] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P 1332353418394447a3543a8890e38e52: Deleting consensus metadata
I20260923 18:05:42.024587 20936 ts_tablet_manager.cc:1918] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:05:42.024688 20936 log.cc:1196] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/5cc9c7c611e74da4b20ac7bf9ae624a7
I20260923 18:05:42.025213 20330 catalog_manager.cc:4841] TS fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869): tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (table test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]) successfully deleted
I20260923 18:05:42.026333 20327 catalog_manager.cc:4841] TS 1332353418394447a3543a8890e38e52 (127.18.44.194:35883): tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (table test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]) successfully deleted
I20260923 18:05:42.027025 20936 ts_tablet_manager.cc:1939] T 5cc9c7c611e74da4b20ac7bf9ae624a7 P f490e6b2302a47d79ba13bcde76bfd01: Deleting consensus metadata
I20260923 18:05:42.028105 20329 catalog_manager.cc:4841] TS f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529): tablet 5cc9c7c611e74da4b20ac7bf9ae624a7 (table test-table [id=d7fa2ee3d4d04fad8acddae8191a4e0b]) successfully deleted
I20260923 18:05:42.031778 20760 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14e48acf9542da9185163c645869bb (DEFAULT_TABLE table=test-table [id=37f6f9428fc541dfa09b1aff54d26ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:42.031782 20620 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14e48acf9542da9185163c645869bb (DEFAULT_TABLE table=test-table [id=37f6f9428fc541dfa09b1aff54d26ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:42.031874 20480 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14e48acf9542da9185163c645869bb (DEFAULT_TABLE table=test-table [id=37f6f9428fc541dfa09b1aff54d26ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:42.032162 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a14e48acf9542da9185163c645869bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:42.032162 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a14e48acf9542da9185163c645869bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:42.032193 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a14e48acf9542da9185163c645869bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:42.034827 20947 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: Bootstrap starting.
I20260923 18:05:42.034946 20946 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Bootstrap starting.
I20260923 18:05:42.035825 20947 tablet_bootstrap.cc:654] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:42.036001 20946 tablet_bootstrap.cc:654] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:42.036728 20948 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Bootstrap starting.
I20260923 18:05:42.037204 20947 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: No bootstrap required, opened a new log
I20260923 18:05:42.037314 20947 ts_tablet_manager.cc:1397] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:42.037658 20948 tablet_bootstrap.cc:654] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:42.037739 20947 raft_consensus.cc:348] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.037868 20947 raft_consensus.cc:374] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:42.037909 20947 raft_consensus.cc:729] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee495ca8b1a4908b71d4e1ed8024aac, State: Initialized, Role: FOLLOWER
I20260923 18:05:42.038028 20947 consensus_queue.cc:260] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.038285 20947 ts_tablet_manager.cc:1428] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:05:42.039000 20948 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: No bootstrap required, opened a new log
I20260923 18:05:42.039116 20948 ts_tablet_manager.cc:1397] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:05:42.039654 20948 raft_consensus.cc:348] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.039794 20948 raft_consensus.cc:374] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:42.039824 20948 raft_consensus.cc:729] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1332353418394447a3543a8890e38e52, State: Initialized, Role: FOLLOWER
I20260923 18:05:42.039994 20946 tablet_bootstrap.cc:492] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: No bootstrap required, opened a new log
I20260923 18:05:42.039981 20948 consensus_queue.cc:260] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.040099 20946 ts_tablet_manager.cc:1397] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260923 18:05:42.040274 20948 ts_tablet_manager.cc:1428] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:05:42.040560 20946 raft_consensus.cc:348] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.040683 20946 raft_consensus.cc:374] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:42.040710 20946 raft_consensus.cc:729] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f490e6b2302a47d79ba13bcde76bfd01, State: Initialized, Role: FOLLOWER
I20260923 18:05:42.040835 20946 consensus_queue.cc:260] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.041121 20946 ts_tablet_manager.cc:1428] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260923 18:05:42.276189 20929 raft_consensus.cc:482] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:42.276420 20929 raft_consensus.cc:504] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.277051 20929 leader_election.cc:288] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:42.277819 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a14e48acf9542da9185163c645869bb" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" is_pre_election: true
I20260923 18:05:42.277819 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a14e48acf9542da9185163c645869bb" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01" is_pre_election: true
I20260923 18:05:42.278038 20765 raft_consensus.cc:2454] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:42.278038 20500 raft_consensus.cc:2454] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 0.
I20260923 18:05:42.278487 20574 leader_election.cc:302] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, f490e6b2302a47d79ba13bcde76bfd01, fee495ca8b1a4908b71d4e1ed8024aac; no voters:
I20260923 18:05:42.278777 20929 raft_consensus.cc:2781] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:42.278844 20929 raft_consensus.cc:482] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:42.278882 20929 raft_consensus.cc:3037] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:42.280222 20929 raft_consensus.cc:504] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.280625 20929 leader_election.cc:288] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Requested vote from peers fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869), f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529)
I20260923 18:05:42.281160 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a14e48acf9542da9185163c645869bb" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee495ca8b1a4908b71d4e1ed8024aac"
I20260923 18:05:42.281315 20500 raft_consensus.cc:3037] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:42.281332 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a14e48acf9542da9185163c645869bb" candidate_uuid: "1332353418394447a3543a8890e38e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f490e6b2302a47d79ba13bcde76bfd01"
I20260923 18:05:42.281566 20765 raft_consensus.cc:3037] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:42.282446 20500 raft_consensus.cc:2454] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:42.282826 20574 leader_election.cc:302] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1332353418394447a3543a8890e38e52, fee495ca8b1a4908b71d4e1ed8024aac; no voters:
I20260923 18:05:42.283004 20929 raft_consensus.cc:2781] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:42.283102 20929 raft_consensus.cc:686] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Becoming Leader. State: Replica: 1332353418394447a3543a8890e38e52, State: Running, Role: LEADER
I20260923 18:05:42.283319 20929 consensus_queue.cc:237] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } }
I20260923 18:05:42.283617 20765 raft_consensus.cc:2454] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1332353418394447a3543a8890e38e52 in term 1.
I20260923 18:05:42.285274 20342 catalog_manager.cc:5495] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1332353418394447a3543a8890e38e52 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "1332353418394447a3543a8890e38e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1332353418394447a3543a8890e38e52" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:42.365039 20500 raft_consensus.cc:1261] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:42.365137 20765 raft_consensus.cc:1261] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Refusing update from remote peer 1332353418394447a3543a8890e38e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:42.365792 20929 consensus_queue.cc:1035] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f490e6b2302a47d79ba13bcde76bfd01" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:42.366014 20911 consensus_queue.cc:1035] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee495ca8b1a4908b71d4e1ed8024aac" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 18:05:42.387032 20917 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 18:05:43.975703 20342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:43.976080 20342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:43.980476 20342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8474f5d9b74be38832613a08440af6: Sending DeleteTablet for 3 replicas of tablet 1a14e48acf9542da9185163c645869bb
I20260923 18:05:43.981395 20480 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a14e48acf9542da9185163c645869bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:43 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48944
I20260923 18:05:43.981411 20620 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a14e48acf9542da9185163c645869bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:43 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48866
I20260923 18:05:43.981613 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a14e48acf9542da9185163c645869bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:43 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40224
I20260923 18:05:43.981875 20988 tablet_replica.cc:330] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: stopping tablet replica
I20260923 18:05:43.982033 20988 raft_consensus.cc:2229] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:43.982155 20990 tablet_replica.cc:330] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: stopping tablet replica
I20260923 18:05:43.982236 20988 raft_consensus.cc:2258] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:43.982227 20989 tablet_replica.cc:330] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: stopping tablet replica
I20260923 18:05:43.982393 20990 raft_consensus.cc:2229] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:43.982529 20990 raft_consensus.cc:2258] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:43.982875 20989 raft_consensus.cc:2229] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:43.983404 20989 raft_consensus.cc:2258] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:43.983523 20990 ts_tablet_manager.cc:1905] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:43.983533 20988 ts_tablet_manager.cc:1905] T 1a14e48acf9542da9185163c645869bb P fee495ca8b1a4908b71d4e1ed8024aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:43.983925 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 20415
I20260923 18:05:43.985582 20989 ts_tablet_manager.cc:1905] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:43.988236 20990 ts_tablet_manager.cc:1918] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:05:43.988476 20990 log.cc:1196] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/1a14e48acf9542da9185163c645869bb
I20260923 18:05:43.988824 20990 ts_tablet_manager.cc:1939] T 1a14e48acf9542da9185163c645869bb P f490e6b2302a47d79ba13bcde76bfd01: Deleting consensus metadata
I20260923 18:05:43.991530 20329 catalog_manager.cc:4841] TS f490e6b2302a47d79ba13bcde76bfd01 (127.18.44.195:36529): tablet 1a14e48acf9542da9185163c645869bb (table test-table [id=37f6f9428fc541dfa09b1aff54d26ade]) successfully deleted
I20260923 18:05:43.992905 20989 ts_tablet_manager.cc:1918] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:05:43.993098 20989 log.cc:1196] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/1a14e48acf9542da9185163c645869bb
I20260923 18:05:43.993487 20989 ts_tablet_manager.cc:1939] T 1a14e48acf9542da9185163c645869bb P 1332353418394447a3543a8890e38e52: Deleting consensus metadata
I20260923 18:05:43.995500 20327 catalog_manager.cc:4841] TS 1332353418394447a3543a8890e38e52 (127.18.44.194:35883): tablet 1a14e48acf9542da9185163c645869bb (table test-table [id=37f6f9428fc541dfa09b1aff54d26ade]) successfully deleted
W20260923 18:05:43.997586 20330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:33869 (error 108)
I20260923 18:05:43.998039 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 20555
I20260923 18:05:44.000133 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:43.999207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33869 (local address 127.0.0.1:56368)
0923 18:05:43.999849 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:43.999967 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":164,"threads_started":1}
W20260923 18:05:44.000387 20330 catalog_manager.cc:4584] TS fee495ca8b1a4908b71d4e1ed8024aac (127.18.44.193:33869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a14e48acf9542da9185163c645869bb: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33869: connect: Connection refused (error 111)
I20260923 18:05:44.005858 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 20695
I20260923 18:05:44.013792 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 20311
2026-09-23T18:05:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7040 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:44 dist-test-slave-rvjj krb5kdc[20995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:44 dist-test-slave-rvjj krb5kdc[20995](info): set up 2 sockets
Sep 23 18:05:44 dist-test-slave-rvjj krb5kdc[20995](info): commencing operation
krb5kdc: starting...
W20260923 18:05:45.036803 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:45.184535 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37121
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:45.304980 21008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:45.305330 21008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:45.305388 21008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:45.309794 21008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:45.309890 21008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:45.309917 21008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:45.309938 21008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:45.309958 21008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:45.315889 21008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:45.317065 21008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:45.318153 21008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:45.329798 21014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:45.329823 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:45.329921 21008 server_base.cc:1034] running on GCE node
W20260923 18:05:45.329823 21013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:45.331043 21008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:45.333123 21008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:45.334410 21008 hybrid_clock.cc:648] HybridClock initialized: now 1790186745334359 us; error 77 us; skew 500 ppm
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:45.338951 21008 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:45.340476 21008 webserver.cc:460] Webserver started at http://127.18.44.254:46323/ using document root <none> and password file <none>
I20260923 18:05:45.341290 21008 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:45.341377 21008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:45.341650 21008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:45.344754 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "ff3c3dc7c57247b49792ed46bc980dc6"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "64de09c3db964d832cfb05455bcaa864"
server_key_iv: "358b4bafbdf592f0c7382ee8a078b8ef"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.345592 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "ff3c3dc7c57247b49792ed46bc980dc6"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "64de09c3db964d832cfb05455bcaa864"
server_key_iv: "358b4bafbdf592f0c7382ee8a078b8ef"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.352331 21008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 18:05:45.356699 21023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.358913 21008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260923 18:05:45.359156 21008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "ff3c3dc7c57247b49792ed46bc980dc6"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "64de09c3db964d832cfb05455bcaa864"
server_key_iv: "358b4bafbdf592f0c7382ee8a078b8ef"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.359447 21008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:45.377254 21008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:45.383074 21008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:45.383463 21008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:45.393781 21008 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37121
I20260923 18:05:45.393852 21075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37121 every 8 connection(s)
I20260923 18:05:45.395182 21008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:45.398746 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:45.404173 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21008
I20260923 18:05:45.404325 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:45.404826 18611 external_mini_cluster.cc:1428] Setting key 4ef423e9f1bc67a906d12f6f71e0824e
I20260923 18:05:45.405294 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6: Bootstrap starting.
I20260923 18:05:45.408394 21076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:45.409471 21076 log.cc:824] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:45.412839 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6: No bootstrap required, opened a new log
I20260923 18:05:45.417244 21076 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } }
I20260923 18:05:45.417529 21076 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:45.417582 21076 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3c3dc7c57247b49792ed46bc980dc6, State: Initialized, Role: FOLLOWER
I20260923 18:05:45.418116 21076 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } }
I20260923 18:05:45.418422 21076 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:45.418520 21076 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:45.418651 21076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:45.420007 21076 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } }
I20260923 18:05:45.420396 21076 leader_election.cc:302] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff3c3dc7c57247b49792ed46bc980dc6; no voters:
I20260923 18:05:45.420773 21076 leader_election.cc:288] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:45.420876 21081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:45.421092 21081 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [term 1 LEADER]: Becoming Leader. State: Replica: ff3c3dc7c57247b49792ed46bc980dc6, State: Running, Role: LEADER
I20260923 18:05:45.421433 21081 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } }
I20260923 18:05:45.422492 21076 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:45.423748 21082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff3c3dc7c57247b49792ed46bc980dc6. Latest consensus state: current_term: 1 leader_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } } }
I20260923 18:05:45.423991 21082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:45.424607 21091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:45.424811 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3c3dc7c57247b49792ed46bc980dc6" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37121 } } }
I20260923 18:05:45.427089 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:45.428501 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:45.406926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.18.44.254:37121)
0923 18:05:45.407661 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:45.407684 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:45.407921 (+ 237us) server_negotiation.cc:407] Connection header received
0923 18:05:45.409016 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:45.409044 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:45.409558 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:45.409990 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:45.411791 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.413306 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.414783 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.415627 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.415866 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:45.419522 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:45.419530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:45.419550 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:45.419595 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:45.422310 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:45.423151 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:45.423160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:45.423168 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:45.423455 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:45.425062 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:45.425067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:45.425068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:45.425807 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:45.426115 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:45.426829 (+ 714us) server_negotiation.cc:299] Negotiation successful
0923 18:05:45.427322 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":225,"threads_started":1}
I20260923 18:05:45.429153 21091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:45.437989 21091 catalog_manager.cc:1347] Generated new cluster ID: 254fec263f184902bd0f26dc4097c9e3
I20260923 18:05:45.438103 21091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:45.456292 21091 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:45.458073 21091 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:45.467223 21091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff3c3dc7c57247b49792ed46bc980dc6: Generated new TSK 0
I20260923 18:05:45.468412 21091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:45.517446 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:45.663918 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:45.664273 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:45.664363 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:45.668926 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:45.669010 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:45.669096 21102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:45.674882 21102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:45.676110 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:45.677275 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:45.689035 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:45.689195 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:45.689244 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:45.689262 21102 server_base.cc:1034] running on GCE node
I20260923 18:05:45.690359 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:45.691145 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:45.692370 21102 hybrid_clock.cc:648] HybridClock initialized: now 1790186745692337 us; error 65 us; skew 500 ppm
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:45.696091 21102 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:45.697268 21102 webserver.cc:460] Webserver started at http://127.18.44.193:38715/ using document root <none> and password file <none>
I20260923 18:05:45.697949 21102 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:45.698019 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:45.698236 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:45.700503 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "71db006245b6443b8003b3d2db6bf4c3"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "ec78afaff7dd8182221510046df5b17b"
server_key_iv: "20fe1b59a25df90849b7c7f3dbbed254"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.701095 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "71db006245b6443b8003b3d2db6bf4c3"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "ec78afaff7dd8182221510046df5b17b"
server_key_iv: "20fe1b59a25df90849b7c7f3dbbed254"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.705628 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:05:45.709231 21117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.711199 21102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 18:05:45.711438 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "71db006245b6443b8003b3d2db6bf4c3"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "ec78afaff7dd8182221510046df5b17b"
server_key_iv: "20fe1b59a25df90849b7c7f3dbbed254"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.711583 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:45.723847 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:45.728302 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:45.728516 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:45.729192 21102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:45.730171 21102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:45.730225 21102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.730281 21102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:45.730310 21102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.741250 21102 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:44365
I20260923 18:05:45.741324 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:44365 every 8 connection(s)
I20260923 18:05:45.742395 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:45.748667 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21102
I20260923 18:05:45.748795 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:45.749073 18611 external_mini_cluster.cc:1428] Setting key c6528585ddf7aba8083f3a2e47df9b51
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:45.760923 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:45.745312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:45537 (local address 127.18.44.254:37121)
0923 18:05:45.745653 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:45.745657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:45.747119 (+ 1462us) server_negotiation.cc:407] Connection header received
0923 18:05:45.748353 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:45.748357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:45.748431 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:45.748520 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:45.750321 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.750929 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.752378 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.752648 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.752751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:45.756330 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:45.756334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:45.756337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:45.756376 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:45.758063 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:45.758903 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:45.758908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:45.758910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:45.759013 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:45.759464 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:45.759467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:45.759468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:45.759726 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:45.759828 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:45.760528 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 18:05:45.760719 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:45.762228 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:45.745649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37121 (local address 127.18.44.193:45537)
0923 18:05:45.746825 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:45.746900 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:45.748051 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:45.748796 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:45.748808 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:45.749397 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:45.750119 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:45.750139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.751078 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.751081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:45.752228 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:45.752242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:45.752838 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:45.752843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:45.753073 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:45.753837 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:45.753859 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:45.756026 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:45.758265 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:45.758274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:45.758298 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:45.758743 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:45.759111 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:45.759114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:45.759115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:45.759354 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:45.759843 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:45.759853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:45.760227 (+ 374us) client_negotiation.cc:769] Sending connection context
0923 18:05:45.760453 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 18:05:45.761026 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":546,"threads_started":1}
I20260923 18:05:45.764101 21231 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37121
I20260923 18:05:45.764482 21231 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:45.765287 21231 heartbeater.cc:511] Master 127.18.44.254:37121 requested a full tablet report, sending...
I20260923 18:05:45.767537 21040 ts_manager.cc:194] Registered new tserver with Master: 71db006245b6443b8003b3d2db6bf4c3 (127.18.44.193:44365)
I20260923 18:05:45.768947 21040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:45537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:45.786351 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:45.910749 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:45.911075 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:45.911121 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:45.916313 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:45.916417 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:45.916484 21236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:45.923727 21236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:45.925025 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:45.926162 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:45.938820 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:45.938743 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:45.939237 21241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:45.939532 21236 server_base.cc:1034] running on GCE node
I20260923 18:05:45.940065 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:45.940872 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:45.942122 21236 hybrid_clock.cc:648] HybridClock initialized: now 1790186745942085 us; error 56 us; skew 500 ppm
Sep 23 18:05:45 dist-test-slave-rvjj krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:45.945363 21236 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:45.946512 21236 webserver.cc:460] Webserver started at http://127.18.44.194:41137/ using document root <none> and password file <none>
I20260923 18:05:45.947151 21236 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:45.947235 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:45.947486 21236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:45.950090 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "5151fb599193456a9213b4c9f19e99ba"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "e7fbafa7ba7d905bd9272b9dfaf61618"
server_key_iv: "5a5d8f883e837e1f4836e0b2c445efe5"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.950892 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "5151fb599193456a9213b4c9f19e99ba"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "e7fbafa7ba7d905bd9272b9dfaf61618"
server_key_iv: "5a5d8f883e837e1f4836e0b2c445efe5"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.956128 21236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:05:45.960695 21251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.963122 21236 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260923 18:05:45.963400 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "5151fb599193456a9213b4c9f19e99ba"
format_stamp: "Formatted at 2026-09-23 18:05:45 on dist-test-slave-rvjj"
server_key: "e7fbafa7ba7d905bd9272b9dfaf61618"
server_key_iv: "5a5d8f883e837e1f4836e0b2c445efe5"
server_key_version: "encryptionkey@0"
I20260923 18:05:45.963554 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:45.983683 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:45.987098 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:45.987342 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:45.988018 21236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:45.988983 21236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:45.989034 21236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.989075 21236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:45.989089 21236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:45.998986 21236 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39631
I20260923 18:05:45.999039 21364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39631 every 8 connection(s)
I20260923 18:05:46.000064 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:46.005085 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21236
I20260923 18:05:46.005201 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:46.005475 18611 external_mini_cluster.cc:1428] Setting key cdd1858d9057ba71f30d01b7d0dc3c32
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[20995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:05:46.020893 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:46.002941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:51999 (local address 127.18.44.254:37121)
0923 18:05:46.003154 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:46.003159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:46.004423 (+ 1264us) server_negotiation.cc:407] Connection header received
0923 18:05:46.006203 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:46.006207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:46.006274 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:46.006386 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:46.008740 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.009590 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.011201 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.011586 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.011723 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:46.015599 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:46.015605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:46.015609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:46.015657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:46.017718 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:46.018542 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:46.018547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:46.018549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:46.018673 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:46.019336 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:46.019341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:46.019343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:46.019715 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:46.019841 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:46.020557 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 18:05:46.020725 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 18:05:46.021859 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:46.003408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37121 (local address 127.18.44.194:51999)
0923 18:05:46.004164 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:46.004243 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:46.005823 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:46.006742 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:46.006756 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:46.007533 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:46.008489 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:46.008518 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.009796 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.009802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:46.011015 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:46.011031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.011868 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.011873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:46.012153 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:46.013126 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:46.013153 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:46.015224 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:46.017944 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:46.017953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:46.017973 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:46.018372 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:46.018889 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:46.018892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:46.018893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:46.019123 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:46.019861 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:46.019867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:46.020193 (+ 326us) client_negotiation.cc:769] Sending connection context
0923 18:05:46.020422 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 18:05:46.020767 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:46.024185 21365 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37121
I20260923 18:05:46.024613 21365 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:46.025302 21365 heartbeater.cc:511] Master 127.18.44.254:37121 requested a full tablet report, sending...
I20260923 18:05:46.026917 21040 ts_manager.cc:194] Registered new tserver with Master: 5151fb599193456a9213b4c9f19e99ba (127.18.44.194:39631)
I20260923 18:05:46.028043 21040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:51999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:46.049315 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:46.215319 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:46.216213 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:46.216423 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:46.223791 21370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:46.223919 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:46.224020 21370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:46.232566 21370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:46.234575 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:46.237301 21370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:46.251592 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:46.251716 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:46.251631 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:46.252020 21370 server_base.cc:1034] running on GCE node
I20260923 18:05:46.252728 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:46.253703 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:46.254947 21370 hybrid_clock.cc:648] HybridClock initialized: now 1790186746254935 us; error 36 us; skew 500 ppm
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186746, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:46.258893 21370 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:46.260474 21370 webserver.cc:460] Webserver started at http://127.18.44.195:45625/ using document root <none> and password file <none>
I20260923 18:05:46.261294 21370 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:46.261389 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:46.261646 21370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:46.265257 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "2b3aafd8c0be4b32a7af7e52283d7f32"
format_stamp: "Formatted at 2026-09-23 18:05:46 on dist-test-slave-rvjj"
server_key: "e2f4387809926f6427c22973e994c80d"
server_key_iv: "b29cb2404402bfb0890016ef3ba991bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:46.266194 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "2b3aafd8c0be4b32a7af7e52283d7f32"
format_stamp: "Formatted at 2026-09-23 18:05:46 on dist-test-slave-rvjj"
server_key: "e2f4387809926f6427c22973e994c80d"
server_key_iv: "b29cb2404402bfb0890016ef3ba991bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:46.272642 21370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:05:46.277913 21385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:46.280201 21370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:46.280452 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "2b3aafd8c0be4b32a7af7e52283d7f32"
format_stamp: "Formatted at 2026-09-23 18:05:46 on dist-test-slave-rvjj"
server_key: "e2f4387809926f6427c22973e994c80d"
server_key_iv: "b29cb2404402bfb0890016ef3ba991bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:46.280637 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:46.293639 21370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:46.298760 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:46.299034 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:46.299903 21370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:46.301051 21370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:46.301110 21370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:46.301173 21370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:46.301200 21370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:46.312896 21370 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:32875
I20260923 18:05:46.313045 21498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:32875 every 8 connection(s)
I20260923 18:05:46.313977 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:46.320262 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21370
I20260923 18:05:46.320398 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:46.320663 18611 external_mini_cluster.cc:1428] Setting key c8de125223b8454e0de80359c3bee227
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[20995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186746, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:46.334519 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:46.316752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:44749 (local address 127.18.44.254:37121)
0923 18:05:46.316921 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:46.316925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:46.318100 (+ 1175us) server_negotiation.cc:407] Connection header received
0923 18:05:46.319824 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:46.319828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:46.319908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:46.320005 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:46.322041 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.322789 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.324976 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.325280 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.325405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:46.329228 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:46.329312 (+ 84us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:46.329316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:46.329374 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:46.331153 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:46.332101 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:46.332108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:46.332110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:46.332227 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:46.332872 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:46.332876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:46.332877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:46.333171 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:46.333282 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:46.334111 (+ 829us) server_negotiation.cc:299] Negotiation successful
0923 18:05:46.334354 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:05:46.335502 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:46.317124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37121 (local address 127.18.44.195:44749)
0923 18:05:46.317822 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:46.317896 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:46.319518 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:46.320354 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:46.320365 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:46.320969 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:46.321872 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:46.321902 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.323005 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.323010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:46.324637 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:46.324670 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:46.325550 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:46.325555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:46.325960 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:46.326723 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:46.326743 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:46.328868 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:46.331430 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:46.331438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:46.331458 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:46.331868 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:46.332352 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:46.332354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:46.332356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:46.332539 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:46.333335 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:46.333342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:46.333702 (+ 360us) client_negotiation.cc:769] Sending connection context
0923 18:05:46.333915 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 18:05:46.334286 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":143,"threads_started":1}
I20260923 18:05:46.336985 21499 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37121
I20260923 18:05:46.337301 21499 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:46.337989 21499 heartbeater.cc:511] Master 127.18.44.254:37121 requested a full tablet report, sending...
I20260923 18:05:46.339659 21040 ts_manager.cc:194] Registered new tserver with Master: 2b3aafd8c0be4b32a7af7e52283d7f32 (127.18.44.195:32875)
I20260923 18:05:46.340382 21040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:44749
I20260923 18:05:46.349705 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:05:46.367301 21079 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:05:46.365252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.18.44.254:37121)
0923 18:05:46.365482 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:46.365488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:46.365894 (+ 406us) server_negotiation.cc:407] Connection header received
0923 18:05:46.366053 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:46.366069 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:46.366123 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:46.366195 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:46.367128 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38584: BlockingRecv error: recv got EOF from 127.0.0.1:38584 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 18:05:46.368721 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21102
I20260923 18:05:46.374408 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21236
I20260923 18:05:46.382057 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21370
I20260923 18:05:46.387691 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21008
2026-09-23T18:05:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2373 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[21511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[21511](info): set up 2 sockets
Sep 23 18:05:46 dist-test-slave-rvjj krb5kdc[21511](info): commencing operation
krb5kdc: starting...
W20260923 18:05:47.404937 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:47.566063 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37931
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:47.718107 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:47.718443 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:47.718487 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:47.722779 21524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:47.722880 21524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:47.722899 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:47.722914 21524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:47.722929 21524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:47.729123 21524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:47.730350 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:47.731482 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:47.741354 21530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:47.741379 21532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:47.741555 21524 server_base.cc:1034] running on GCE node
W20260923 18:05:47.741398 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:47.742575 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:47.744097 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:47.745325 21524 hybrid_clock.cc:648] HybridClock initialized: now 1790186747745277 us; error 69 us; skew 500 ppm
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186747, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:47.749591 21524 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:47.750825 21524 webserver.cc:460] Webserver started at http://127.18.44.254:40495/ using document root <none> and password file <none>
I20260923 18:05:47.751528 21524 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:47.751603 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:47.751825 21524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:47.754148 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "4569fc03136548eab41aafd53b73ef8f"
format_stamp: "Formatted at 2026-09-23 18:05:47 on dist-test-slave-rvjj"
server_key: "58e5930999c7f7d9712f7a8128db244a"
server_key_iv: "bd06f92b4c02d13ee63fa3dab8b8a578"
server_key_version: "encryptionkey@0"
I20260923 18:05:47.754748 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "4569fc03136548eab41aafd53b73ef8f"
format_stamp: "Formatted at 2026-09-23 18:05:47 on dist-test-slave-rvjj"
server_key: "58e5930999c7f7d9712f7a8128db244a"
server_key_iv: "bd06f92b4c02d13ee63fa3dab8b8a578"
server_key_version: "encryptionkey@0"
I20260923 18:05:47.759083 21524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260923 18:05:47.761950 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:47.763610 21524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:47.763787 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "4569fc03136548eab41aafd53b73ef8f"
format_stamp: "Formatted at 2026-09-23 18:05:47 on dist-test-slave-rvjj"
server_key: "58e5930999c7f7d9712f7a8128db244a"
server_key_iv: "bd06f92b4c02d13ee63fa3dab8b8a578"
server_key_version: "encryptionkey@0"
I20260923 18:05:47.763916 21524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:47.790722 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:47.794467 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:47.794675 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:47.802104 21524 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37931
I20260923 18:05:47.802449 21591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37931 every 8 connection(s)
I20260923 18:05:47.803226 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:47.805460 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21524
I20260923 18:05:47.805614 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:47.805951 18611 external_mini_cluster.cc:1428] Setting key 72cfb923b3edddf35b0550ab02f10e60
I20260923 18:05:47.806847 21592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:47.814462 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: Bootstrap starting.
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:47.817304 21592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:47.818284 21592 log.cc:824] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:47.820865 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: No bootstrap required, opened a new log
I20260923 18:05:47.822861 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:47.807619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.18.44.254:37931)
0923 18:05:47.808134 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:47.808159 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:47.808208 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:05:47.809110 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:47.809151 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:47.809629 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:47.810079 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:47.810921 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:47.812496 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:47.813520 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:47.814287 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:47.814469 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:47.817311 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:47.817320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:47.817340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:47.817384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:47.819566 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:47.820179 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:47.820185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:47.820190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:47.820323 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:47.820683 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:47.820685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:47.820686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:47.821170 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:47.821342 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:47.821706 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 18:05:47.822063 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":140,"threads_started":1}
I20260923 18:05:47.825528 21592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } }
I20260923 18:05:47.825876 21592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:47.825940 21592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4569fc03136548eab41aafd53b73ef8f, State: Initialized, Role: FOLLOWER
I20260923 18:05:47.826411 21592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } }
I20260923 18:05:47.826562 21592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:47.826638 21592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:47.826681 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:05:47.826723 21592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:47.828047 21592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } }
I20260923 18:05:47.828425 21592 leader_election.cc:302] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4569fc03136548eab41aafd53b73ef8f; no voters:
I20260923 18:05:47.828786 21592 leader_election.cc:288] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:47.828881 21598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:47.829069 21598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [term 1 LEADER]: Becoming Leader. State: Replica: 4569fc03136548eab41aafd53b73ef8f, State: Running, Role: LEADER
I20260923 18:05:47.830021 21598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } }
I20260923 18:05:47.830355 21592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:47.831822 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4569fc03136548eab41aafd53b73ef8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } } }
I20260923 18:05:47.831801 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4569fc03136548eab41aafd53b73ef8f. Latest consensus state: current_term: 1 leader_uuid: "4569fc03136548eab41aafd53b73ef8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4569fc03136548eab41aafd53b73ef8f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37931 } } }
I20260923 18:05:47.831984 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:47.831984 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:47.836144 21613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:47.836287 21613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:47.836422 21609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:47.837468 21609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:47.844246 21609 catalog_manager.cc:1347] Generated new cluster ID: 7c74fab94ac043238a9c5c11e496ccb6
I20260923 18:05:47.844340 21609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:47.849277 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:47.844112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.18.44.254:37931)
0923 18:05:47.844232 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:47.844245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:47.844814 (+ 569us) server_negotiation.cc:407] Connection header received
0923 18:05:47.844941 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:47.844944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:47.844992 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:47.845058 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:47.845623 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:47.846168 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:47.847037 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:47.847357 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:47.847453 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:47.848302 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:47.848306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:47.848309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:47.848794 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:47.848830 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:47.848848 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:47.848934 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:47.849038 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 18:05:47.849112 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:05:47.874922 21609 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:47.875991 21609 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:47.887611 21609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4569fc03136548eab41aafd53b73ef8f: Generated new TSK 0
I20260923 18:05:47.888609 21609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:05:47.897976 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21524
2026-09-23T18:05:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1505 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21625](info): set up 2 sockets
Sep 23 18:05:47 dist-test-slave-rvjj krb5kdc[21625](info): commencing operation
krb5kdc: starting...
W20260923 18:05:48.911520 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:49.052536 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36239
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:49.174069 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:49.174386 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:49.174429 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:49.178737 21638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:49.178817 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:49.178849 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:49.178866 21638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:49.178880 21638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:49.184592 21638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:49.185694 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:49.186774 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:49.195726 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:49.195905 21643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.195830 21638 server_base.cc:1034] running on GCE node
W20260923 18:05:49.196115 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.196713 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:49.197917 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:49.199115 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790186749199068 us; error 61 us; skew 500 ppm
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:49.206251 21638 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:49.207397 21638 webserver.cc:460] Webserver started at http://127.18.44.254:43735/ using document root <none> and password file <none>
I20260923 18:05:49.207996 21638 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:49.208069 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:49.208266 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:49.210448 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "e4f22728f7dd462eb4393e26257b6cac"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "110e7ce30255159807260a8fe4dbc53a"
server_key_iv: "27d837b5bbfed57a3e6ad8e53d24249d"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.211037 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "e4f22728f7dd462eb4393e26257b6cac"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "110e7ce30255159807260a8fe4dbc53a"
server_key_iv: "27d837b5bbfed57a3e6ad8e53d24249d"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.215153 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:05:49.217864 21653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.219321 21638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:49.219458 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "e4f22728f7dd462eb4393e26257b6cac"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "110e7ce30255159807260a8fe4dbc53a"
server_key_iv: "27d837b5bbfed57a3e6ad8e53d24249d"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.219573 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:49.251686 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:49.254820 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:49.255030 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:49.262398 21638 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:36239
I20260923 18:05:49.262722 21705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:36239 every 8 connection(s)
I20260923 18:05:49.263492 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:49.266402 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:49.270177 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21638
I20260923 18:05:49.270301 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:49.270568 18611 external_mini_cluster.cc:1428] Setting key 3b2456c9287f3fb22d0c20a5cef1ef10
I20260923 18:05:49.272940 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac: Bootstrap starting.
I20260923 18:05:49.275312 21706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:49.276099 21706 log.cc:824] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:49.278167 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac: No bootstrap required, opened a new log
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:49.282174 21706 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } }
I20260923 18:05:49.282505 21706 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:49.282554 21706 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f22728f7dd462eb4393e26257b6cac, State: Initialized, Role: FOLLOWER
I20260923 18:05:49.283097 21706 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } }
I20260923 18:05:49.283245 21706 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:49.283331 21706 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:49.283450 21706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:49.284695 21706 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } }
I20260923 18:05:49.285092 21706 leader_election.cc:302] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4f22728f7dd462eb4393e26257b6cac; no voters:
I20260923 18:05:49.285486 21706 leader_election.cc:288] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:49.285580 21711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:49.285797 21711 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [term 1 LEADER]: Becoming Leader. State: Replica: e4f22728f7dd462eb4393e26257b6cac, State: Running, Role: LEADER
I20260923 18:05:49.286132 21711 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } }
I20260923 18:05:49.286579 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:49.272043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.18.44.254:36239)
0923 18:05:49.272543 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:49.272558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:49.272596 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:05:49.273283 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:49.273302 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:49.273655 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:49.274029 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:49.274834 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.275948 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.276813 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.277293 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.277444 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:49.280312 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:49.280320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:49.280341 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:49.280380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:49.282919 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.283530 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.283537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.283545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.283701 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.284032 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.284035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.284038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.284679 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:49.284894 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:49.285239 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 18:05:49.285571 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":175,"threads_started":1}
I20260923 18:05:49.286847 21706 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:49.287704 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f22728f7dd462eb4393e26257b6cac. Latest consensus state: current_term: 1 leader_uuid: "e4f22728f7dd462eb4393e26257b6cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } } }
I20260923 18:05:49.287813 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4f22728f7dd462eb4393e26257b6cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f22728f7dd462eb4393e26257b6cac" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36239 } } }
I20260923 18:05:49.287884 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:49.287843 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:49.289002 21719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:49.291756 21719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:49.297088 21719 catalog_manager.cc:1347] Generated new cluster ID: 3481538bee784f38b7f5978290da3775
I20260923 18:05:49.297161 21719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:49.306025 21719 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:49.307399 21719 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:49.311488 21719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4f22728f7dd462eb4393e26257b6cac: Generated new TSK 0
I20260923 18:05:49.312253 21719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:49.348124 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:49.470671 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:49.470993 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:49.471056 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:49.475332 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:49.475415 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:49.475499 21732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:49.481137 21732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:49.482344 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:49.483454 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:49.493911 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:49.493919 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.494062 21732 server_base.cc:1034] running on GCE node
W20260923 18:05:49.494334 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.494769 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:49.495525 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:49.496723 21732 hybrid_clock.cc:648] HybridClock initialized: now 1790186749496705 us; error 31 us; skew 500 ppm
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:49.500029 21732 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:49.501019 21732 webserver.cc:460] Webserver started at http://127.18.44.193:38943/ using document root <none> and password file <none>
I20260923 18:05:49.501581 21732 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:49.501658 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:49.501854 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:49.504098 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "71068b9003bc494aa0580ed1bf381dbd"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "86c5bc988c5750ebca2fac61dd98c2d0"
server_key_iv: "b0fd07b19d9c796f19289cf1d4ce5128"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.504690 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "71068b9003bc494aa0580ed1bf381dbd"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "86c5bc988c5750ebca2fac61dd98c2d0"
server_key_iv: "b0fd07b19d9c796f19289cf1d4ce5128"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.508872 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:49.512007 21747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.513378 21732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:05:49.513510 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "71068b9003bc494aa0580ed1bf381dbd"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "86c5bc988c5750ebca2fac61dd98c2d0"
server_key_iv: "b0fd07b19d9c796f19289cf1d4ce5128"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.513648 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:49.545995 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:49.550033 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:49.550239 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:49.550913 21732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:49.551883 21732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:49.551939 21732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.551985 21732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:49.552016 21732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.561326 21732 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:42255
I20260923 18:05:49.561347 21860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:42255 every 8 connection(s)
I20260923 18:05:49.562314 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:49.566474 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21732
I20260923 18:05:49.566699 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:49.567044 18611 external_mini_cluster.cc:1428] Setting key acef96b2a67d7ac1e005864bf7b2e8fa
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:49.578639 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:49.564458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:40191 (local address 127.18.44.254:36239)
0923 18:05:49.564634 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:49.564638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:49.565553 (+ 915us) server_negotiation.cc:407] Connection header received
0923 18:05:49.566964 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:49.566968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:49.567033 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:49.567161 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:49.568750 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.569499 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.571159 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.571477 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.571566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:49.574535 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:49.574538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:49.574540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:49.574568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:49.576055 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.576678 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.576680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.576682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.576776 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.577236 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.577239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.577240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.577477 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:49.577564 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:49.578366 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 18:05:49.578500 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:05:49.579607 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:49.564813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36239 (local address 127.18.44.193:40191)
0923 18:05:49.565371 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:49.565426 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:49.566696 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:49.567458 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:49.567465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:49.567850 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:49.568532 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:49.568564 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.569648 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.569651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:49.571007 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:49.571020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.571642 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.571647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:49.571848 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:49.572551 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:49.572573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:49.574331 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:49.576198 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:49.576205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:49.576224 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:49.576562 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:49.576902 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:49.576906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:49.576909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:49.577131 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:49.577624 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:49.577631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:49.578078 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 18:05:49.578292 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 18:05:49.578610 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:49.581287 21861 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36239
I20260923 18:05:49.581575 21861 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:49.582131 21861 heartbeater.cc:511] Master 127.18.44.254:36239 requested a full tablet report, sending...
I20260923 18:05:49.583963 21670 ts_manager.cc:194] Registered new tserver with Master: 71068b9003bc494aa0580ed1bf381dbd (127.18.44.193:42255)
I20260923 18:05:49.585543 21670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:40191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:49.602890 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:49.720780 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:49.721084 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:49.721127 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:49.725497 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:49.725584 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:49.725648 21866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:49.731333 21866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:49.732414 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:49.733397 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:49.743156 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:49.743178 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:49.743165 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.743479 21866 server_base.cc:1034] running on GCE node
I20260923 18:05:49.743974 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:49.744639 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:49.745832 21866 hybrid_clock.cc:648] HybridClock initialized: now 1790186749745815 us; error 37 us; skew 500 ppm
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:49.748878 21866 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:49.750011 21866 webserver.cc:460] Webserver started at http://127.18.44.194:40341/ using document root <none> and password file <none>
I20260923 18:05:49.750581 21866 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:49.750654 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:49.750837 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:49.753144 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "034260ec8cbc4f1e8dd1a9dac057a2d8"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "8bfb45e1408106095ec435f223353383"
server_key_iv: "ed04c3d39007e7d7ddc796f1231c1783"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.753777 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "034260ec8cbc4f1e8dd1a9dac057a2d8"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "8bfb45e1408106095ec435f223353383"
server_key_iv: "ed04c3d39007e7d7ddc796f1231c1783"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.758546 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 18:05:49.761615 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.763159 21866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:49.763339 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "034260ec8cbc4f1e8dd1a9dac057a2d8"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "8bfb45e1408106095ec435f223353383"
server_key_iv: "ed04c3d39007e7d7ddc796f1231c1783"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.763485 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:49.788162 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:49.791998 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:49.792181 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:49.792773 21866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:49.793742 21866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:49.793797 21866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.793855 21866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:49.793884 21866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:49.803637 21866 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:42897
I20260923 18:05:49.803687 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:42897 every 8 connection(s)
I20260923 18:05:49.804656 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:49.810180 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 21866
I20260923 18:05:49.810319 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:49.810600 18611 external_mini_cluster.cc:1428] Setting key a1d16fcb6aab2c2374ee1fd8091f19a9
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:05:49.821214 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:49.806753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:48315 (local address 127.18.44.254:36239)
0923 18:05:49.806894 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:49.806897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:49.807918 (+ 1021us) server_negotiation.cc:407] Connection header received
0923 18:05:49.809340 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:49.809344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:49.809408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:49.809493 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:49.811023 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.811839 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.813328 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.813649 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.813775 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:49.816860 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:49.816863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:49.816865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:49.816893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:49.818408 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.819071 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.819076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.819078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.819193 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:49.819794 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:49.819797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:49.819799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:49.820078 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:49.820206 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:49.820883 (+ 677us) server_negotiation.cc:299] Negotiation successful
0923 18:05:49.821039 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:05:49.822319 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:49.807104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36239 (local address 127.18.44.194:48315)
0923 18:05:49.807712 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:49.807759 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:49.809100 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:49.809688 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:49.809696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:49.810109 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:49.810678 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:49.810700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.812023 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.812027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:49.813199 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:49.813212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:49.813831 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:49.813834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:49.814000 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:49.814769 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:49.814789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:49.816628 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:49.818581 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:49.818587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:49.818603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:49.818918 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:49.819394 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:49.819397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:49.819399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:49.819601 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:49.820231 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:49.820237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:49.820588 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 18:05:49.820789 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 18:05:49.821175 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":201,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:49.824330 21995 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36239
I20260923 18:05:49.824642 21995 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:49.825350 21995 heartbeater.cc:511] Master 127.18.44.254:36239 requested a full tablet report, sending...
I20260923 18:05:49.826721 21670 ts_manager.cc:194] Registered new tserver with Master: 034260ec8cbc4f1e8dd1a9dac057a2d8 (127.18.44.194:42897)
I20260923 18:05:49.827564 21670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:48315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:49.848034 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:49.966449 22000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:49.966776 22000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:49.966856 22000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:49.971175 22000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:49.971305 22000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:49.971391 22000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:49.977066 22000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:49.978221 22000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:49.979300 22000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:49.989740 22006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:49.989826 22005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.989830 22000 server_base.cc:1034] running on GCE node
W20260923 18:05:49.989750 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:49.990368 22000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:49.991099 22000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:49.992277 22000 hybrid_clock.cc:648] HybridClock initialized: now 1790186749992249 us; error 49 us; skew 500 ppm
Sep 23 18:05:49 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:49.995276 22000 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:49.996238 22000 webserver.cc:460] Webserver started at http://127.18.44.195:37101/ using document root <none> and password file <none>
I20260923 18:05:49.996794 22000 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:49.996855 22000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:49.997030 22000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:49.999245 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "c85a01b0d3c443268068b3e4f5bd7022"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "39516c2a066bf45aab4ab50d7fa4dc9d"
server_key_iv: "00cafa3d8f1ddf6287b9b2a16b870a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:49.999971 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "c85a01b0d3c443268068b3e4f5bd7022"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "39516c2a066bf45aab4ab50d7fa4dc9d"
server_key_iv: "00cafa3d8f1ddf6287b9b2a16b870a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:50.004453 22000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:50.007831 22015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:50.009655 22000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:05:50.009948 22000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "c85a01b0d3c443268068b3e4f5bd7022"
format_stamp: "Formatted at 2026-09-23 18:05:49 on dist-test-slave-rvjj"
server_key: "39516c2a066bf45aab4ab50d7fa4dc9d"
server_key_iv: "00cafa3d8f1ddf6287b9b2a16b870a75"
server_key_version: "encryptionkey@0"
I20260923 18:05:50.010118 22000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:50.038728 22000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:50.042385 22000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:50.042572 22000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:50.043201 22000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:50.044123 22000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:50.044173 22000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:50.044212 22000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:50.044235 22000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:50.053999 22000 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:41837
I20260923 18:05:50.054118 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:41837 every 8 connection(s)
I20260923 18:05:50.054986 22000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:50.055622 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22000
I20260923 18:05:50.055757 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:50.056058 18611 external_mini_cluster.cc:1428] Setting key 137b46002c41de7081609f27558ef6b7
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186749, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:50.074407 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:50.057664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:46089 (local address 127.18.44.254:36239)
0923 18:05:50.057810 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:50.057815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:50.058968 (+ 1153us) server_negotiation.cc:407] Connection header received
0923 18:05:50.060500 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:50.060504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:50.060600 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:50.060944 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:50.062741 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.063441 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.064770 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.064995 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.065090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:50.068470 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:50.068474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:50.068476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:50.068508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:50.071392 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:50.072415 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:50.072419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:50.072420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:50.072525 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:50.073053 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:50.073057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:50.073059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:50.073289 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:50.073395 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:50.074115 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 18:05:50.074272 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:05:50.075362 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:50.058151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36239 (local address 127.18.44.195:46089)
0923 18:05:50.058801 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:50.058854 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:50.060224 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:50.061202 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:50.061212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:50.061684 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:50.062438 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:50.062460 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.063599 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.063602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:50.064601 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:50.064615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.065169 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.065174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:50.065359 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:50.066083 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:50.066102 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:50.068187 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:50.071594 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:50.071602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:50.071625 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:50.072237 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:50.072712 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:50.072715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:50.072716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:50.072909 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:50.073446 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:50.073453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:50.073830 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 18:05:50.074018 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 18:05:50.074341 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":177,"threads_started":1}
I20260923 18:05:50.076900 22129 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36239
I20260923 18:05:50.077207 22129 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:50.077782 22129 heartbeater.cc:511] Master 127.18.44.254:36239 requested a full tablet report, sending...
I20260923 18:05:50.079051 21670 ts_manager.cc:194] Registered new tserver with Master: c85a01b0d3c443268068b3e4f5bd7022 (127.18.44.195:41837)
I20260923 18:05:50.079733 21670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:46089
I20260923 18:05:50.085325 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186750, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186750, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:50.112591 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:50.103431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.18.44.254:36239)
0923 18:05:50.103628 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:50.103633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:50.103725 (+ 92us) server_negotiation.cc:407] Connection header received
0923 18:05:50.103882 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:50.103886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:50.103931 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:50.104009 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:50.104764 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.105319 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.106202 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:50.106478 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:50.106579 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:50.109397 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:50.109401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:50.109403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:50.109437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:50.110816 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:50.111282 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:50.111286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:50.111288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:50.111379 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:50.111638 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:50.111642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:50.111643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:50.111907 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:50.112036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:50.112286 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 18:05:50.112413 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260923 18:05:50.113188 21670 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58822
I20260923 18:05:50.114336 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21732
I20260923 18:05:50.120127 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21866
I20260923 18:05:50.125092 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22000
I20260923 18:05:50.130247 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 21638
2026-09-23T18:05:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2233 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[22141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[22141](info): set up 2 sockets
Sep 23 18:05:50 dist-test-slave-rvjj krb5kdc[22141](info): commencing operation
krb5kdc: starting...
W20260923 18:05:51.140544 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:51 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:51.301259 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45043
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:51.456763 22154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:51.457332 22154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:51.457418 22154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:51.462543 22154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:51.462668 22154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:51.462689 22154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:51.462706 22154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:51.462723 22154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:51.469527 22154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:51.471071 22154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:51.472752 22154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:05:51.485667 22154 server_base.cc:1034] running on GCE node
W20260923 18:05:51.485464 22160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:51.485476 22159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:51.485486 22162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:51.486598 22154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:51.487977 22154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:51.489203 22154 hybrid_clock.cc:648] HybridClock initialized: now 1790186751489186 us; error 48 us; skew 500 ppm
Sep 23 18:05:51 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:51.493114 22154 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:51.494375 22154 webserver.cc:460] Webserver started at http://127.18.44.254:36523/ using document root <none> and password file <none>
I20260923 18:05:51.495030 22154 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:51.495128 22154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:51.495455 22154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:51.497850 22154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "b0ff34df85ac30b06117d821c229c5f8"
server_key_iv: "d29e3627583e45ec8b87de426f620714"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.498485 22154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "b0ff34df85ac30b06117d821c229c5f8"
server_key_iv: "d29e3627583e45ec8b87de426f620714"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.503451 22154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:05:51.506958 22169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:51.509127 22154 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:51.509321 22154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "b0ff34df85ac30b06117d821c229c5f8"
server_key_iv: "d29e3627583e45ec8b87de426f620714"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.509478 22154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:51.540055 22154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:51.543923 22154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:51.544131 22154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:51.552414 22154 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45043
I20260923 18:05:51.552610 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45043 every 8 connection(s)
I20260923 18:05:51.553587 22154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:51.557678 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:51.561128 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22154
I20260923 18:05:51.561293 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:51.561576 18611 external_mini_cluster.cc:1428] Setting key 9ad51ef5af861a9a4b3df20be803efd2
I20260923 18:05:51.567191 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86: Bootstrap starting.
I20260923 18:05:51.570896 22222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:51.572147 22222 log.cc:824] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:05:51 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:51.575932 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86: No bootstrap required, opened a new log
I20260923 18:05:51.581696 22222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } }
I20260923 18:05:51.582086 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:51.563575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.18.44.254:45043)
0923 18:05:51.564106 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:51.564127 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:51.564177 (+ 50us) server_negotiation.cc:407] Connection header received
0923 18:05:51.565414 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:51.565453 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:51.566177 (+ 724us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:51.566719 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:51.567603 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.569250 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.570432 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.571299 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.571538 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:51.574970 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:51.574979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:51.575003 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:51.575117 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:51.577484 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:51.578146 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:51.578154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:51.578163 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:51.578365 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:51.578681 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:51.578685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:51.578687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:51.579519 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:51.579755 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:51.580111 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 18:05:51.580561 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":154,"threads_started":1}
I20260923 18:05:51.582129 22222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:51.582181 22222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3160b2a1445309d2e8c5b3e2dcc86, State: Initialized, Role: FOLLOWER
I20260923 18:05:51.582808 22222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } }
I20260923 18:05:51.582983 22222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:51.583042 22222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:51.583166 22222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:51.584574 22222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } }
I20260923 18:05:51.585070 22222 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca3160b2a1445309d2e8c5b3e2dcc86; no voters:
I20260923 18:05:51.585729 22222 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:51.585858 22227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:51.586203 22227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca3160b2a1445309d2e8c5b3e2dcc86, State: Running, Role: LEADER
I20260923 18:05:51.586651 22227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } }
I20260923 18:05:51.587806 22222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:51.588737 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca3160b2a1445309d2e8c5b3e2dcc86. Latest consensus state: current_term: 1 leader_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } } }
I20260923 18:05:51.588976 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:51.588737 22228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3160b2a1445309d2e8c5b3e2dcc86" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45043 } } }
I20260923 18:05:51.589339 22228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:51.589516 22236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:51.593328 22236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:51.600026 22236 catalog_manager.cc:1347] Generated new cluster ID: 032419c9164f42a2971ea708a6702154
I20260923 18:05:51.600131 22236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:51.609856 22236 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:51.611102 22236 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:51.618559 22236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca3160b2a1445309d2e8c5b3e2dcc86: Generated new TSK 0
I20260923 18:05:51.619367 22236 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:51.660718 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:51.794371 22248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:51.794679 22248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:51.794725 22248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:51.798990 22248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:51.799073 22248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:51.799134 22248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:51.804843 22248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:51.806195 22248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:51.807330 22248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:51.817744 22254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:51.817737 22256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:51.817870 22253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:51.818161 22248 server_base.cc:1034] running on GCE node
I20260923 18:05:51.818632 22248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:51.819334 22248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:51.820513 22248 hybrid_clock.cc:648] HybridClock initialized: now 1790186751820483 us; error 48 us; skew 500 ppm
Sep 23 18:05:51 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:51.823597 22248 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:51.824669 22248 webserver.cc:460] Webserver started at http://127.18.44.193:42503/ using document root <none> and password file <none>
I20260923 18:05:51.825270 22248 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:51.825348 22248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:51.825555 22248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:51.827862 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "3872d97eb27540c79bfab3e727300ece"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "6d9e96d7b882c459daf915400479f947"
server_key_iv: "792a19101f8fb2318ebf32a9018fbb65"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.828471 22248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "3872d97eb27540c79bfab3e727300ece"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "6d9e96d7b882c459daf915400479f947"
server_key_iv: "792a19101f8fb2318ebf32a9018fbb65"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.832839 22248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:51.836228 22263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:51.837792 22248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:51.837940 22248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "3872d97eb27540c79bfab3e727300ece"
format_stamp: "Formatted at 2026-09-23 18:05:51 on dist-test-slave-rvjj"
server_key: "6d9e96d7b882c459daf915400479f947"
server_key_iv: "792a19101f8fb2318ebf32a9018fbb65"
server_key_version: "encryptionkey@0"
I20260923 18:05:51.838073 22248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:51.870328 22248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:51.873785 22248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:51.874001 22248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:51.874655 22248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:51.875609 22248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:51.875686 22248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:51.875877 22248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:51.875941 22248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:51.885401 22248 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:36585
I20260923 18:05:51.885447 22376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:36585 every 8 connection(s)
I20260923 18:05:51.886319 22248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:51.889066 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22248
I20260923 18:05:51.889184 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:51.889524 18611 external_mini_cluster.cc:1428] Setting key 47b4bcfd92a8ee73f0d33f6a2e53d36d
Sep 23 18:05:51 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:51.902693 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:51.888380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:43911 (local address 127.18.44.254:45043)
0923 18:05:51.888551 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:51.888557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:51.889496 (+ 939us) server_negotiation.cc:407] Connection header received
0923 18:05:51.891343 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:51.891348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:51.891432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:51.891563 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:51.893232 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.893788 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.894998 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.895326 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.895437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:51.898201 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:51.898206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:51.898210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:51.898242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:51.899976 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:51.900634 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:51.900638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:51.900640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:51.900742 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:51.901251 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:51.901254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:51.901256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:51.901478 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:51.901566 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:51.902410 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 18:05:51.902553 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:05:51.903837 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:51.888787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45043 (local address 127.18.44.193:43911)
0923 18:05:51.889315 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:51.889372 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:51.891024 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:51.891981 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:51.891994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:51.892444 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:51.893036 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:51.893053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.893932 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.893935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:51.894876 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:51.894890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:51.895498 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:51.895502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:51.895670 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:51.896274 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:51.896292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:51.897969 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:51.900160 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:51.900167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:51.900183 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:51.900513 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:51.900891 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:51.900895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:51.900896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:51.901069 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:51.901702 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:51.901709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:51.902091 (+ 382us) client_negotiation.cc:769] Sending connection context
0923 18:05:51.902330 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 18:05:51.902652 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:51.905336 22377 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45043
I20260923 18:05:51.905603 22377 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:51.906134 22377 heartbeater.cc:511] Master 127.18.44.254:45043 requested a full tablet report, sending...
I20260923 18:05:51.908070 22182 ts_manager.cc:194] Registered new tserver with Master: 3872d97eb27540c79bfab3e727300ece (127.18.44.193:36585)
I20260923 18:05:51.909442 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:43911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:51.925297 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:52.043329 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:52.043653 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:52.043699 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:52.047981 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:52.048063 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:52.048125 22382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:52.053946 22382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:52.055120 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:52.056316 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:52.067229 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:52.067370 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:52.067229 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:52.067641 22382 server_base.cc:1034] running on GCE node
I20260923 18:05:52.068221 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:52.069020 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:52.070225 22382 hybrid_clock.cc:648] HybridClock initialized: now 1790186752070193 us; error 48 us; skew 500 ppm
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:52.073632 22382 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:52.074833 22382 webserver.cc:460] Webserver started at http://127.18.44.194:37055/ using document root <none> and password file <none>
I20260923 18:05:52.075501 22382 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:52.075590 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:52.075924 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:52.078238 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "48b217e9cb1644aa941f2457343bf9c8"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "af4be404533b5dc27738ceaa5302bdce"
server_key_iv: "346e85879d25bce275e66673f8d997bd"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.078814 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "48b217e9cb1644aa941f2457343bf9c8"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "af4be404533b5dc27738ceaa5302bdce"
server_key_iv: "346e85879d25bce275e66673f8d997bd"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.082958 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:05:52.085891 22397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.087484 22382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:52.087632 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "48b217e9cb1644aa941f2457343bf9c8"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "af4be404533b5dc27738ceaa5302bdce"
server_key_iv: "346e85879d25bce275e66673f8d997bd"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.087757 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:52.105175 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:52.108764 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:52.108982 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:52.109653 22382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:52.110776 22382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:52.110844 22382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.110912 22382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:52.110940 22382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.120796 22382 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39861
I20260923 18:05:52.120847 22510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39861 every 8 connection(s)
I20260923 18:05:52.121793 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:52.122846 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22382
I20260923 18:05:52.122987 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:52.123301 18611 external_mini_cluster.cc:1428] Setting key 8561ce2e791177e85d12e480792897e4
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:52.144164 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.124179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:52789 (local address 127.18.44.254:45043)
0923 18:05:52.124312 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:52.124316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:52.125541 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 18:05:52.127419 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:52.127423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:52.127486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:52.127563 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:52.128944 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.129897 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.132206 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.132574 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.132715 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.139185 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:52.139189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:52.139192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:52.139224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:52.141124 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.141928 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.141933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.141935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.142041 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.142560 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.142564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.142566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.142783 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:52.142868 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:52.143876 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0923 18:05:52.144034 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:05:52.145444 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.124622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45043 (local address 127.18.44.194:52789)
0923 18:05:52.125348 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:52.125412 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:52.127077 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:52.127760 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:52.127771 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:52.128185 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:52.128734 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:52.128751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.130158 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.130161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:52.131990 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:52.132009 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.132821 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.132825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:52.133123 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.133800 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:52.133899 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:52.138893 (+ 4994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:52.141355 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:52.141363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:52.141382 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:52.141764 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:52.142209 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:52.142213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:52.142215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:52.142429 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:52.142908 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:52.142917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:52.143418 (+ 501us) client_negotiation.cc:769] Sending connection context
0923 18:05:52.143697 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 18:05:52.144240 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":170,"threads_started":1}
I20260923 18:05:52.147010 22511 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45043
I20260923 18:05:52.147297 22511 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:52.147855 22511 heartbeater.cc:511] Master 127.18.44.254:45043 requested a full tablet report, sending...
I20260923 18:05:52.149292 22182 ts_manager.cc:194] Registered new tserver with Master: 48b217e9cb1644aa941f2457343bf9c8 (127.18.44.194:39861)
I20260923 18:05:52.150018 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:52789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:52.168270 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:52.306982 22516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:52.307354 22516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:52.307416 22516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:52.311939 22516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:52.312052 22516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:52.312119 22516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:52.320001 22516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:52.321208 22516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:52.322299 22516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:52.336182 22521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:52.336181 22524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:52.336205 22522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:52.336464 22516 server_base.cc:1034] running on GCE node
I20260923 18:05:52.337275 22516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:52.338133 22516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:52.339403 22516 hybrid_clock.cc:648] HybridClock initialized: now 1790186752339373 us; error 54 us; skew 500 ppm
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:52.342819 22516 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:52.344025 22516 webserver.cc:460] Webserver started at http://127.18.44.195:39663/ using document root <none> and password file <none>
I20260923 18:05:52.344650 22516 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:52.344735 22516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:52.344962 22516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:52.347699 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "2f8e301322334065aa0e7d90f2c97248"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "786817bf891c2fe836a1d9e7d1d11066"
server_key_iv: "c6a799efd9579267073f8569b17c9248"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.348569 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "2f8e301322334065aa0e7d90f2c97248"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "786817bf891c2fe836a1d9e7d1d11066"
server_key_iv: "c6a799efd9579267073f8569b17c9248"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.354023 22516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:05:52.357659 22531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.359772 22516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 18:05:52.359964 22516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "2f8e301322334065aa0e7d90f2c97248"
format_stamp: "Formatted at 2026-09-23 18:05:52 on dist-test-slave-rvjj"
server_key: "786817bf891c2fe836a1d9e7d1d11066"
server_key_iv: "c6a799efd9579267073f8569b17c9248"
server_key_version: "encryptionkey@0"
I20260923 18:05:52.360098 22516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:52.382023 22516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:52.385962 22516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:52.386199 22516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:52.386864 22516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:52.388435 22516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:52.388504 22516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.388567 22516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:52.388602 22516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:52.400228 22516 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:45805
I20260923 18:05:52.400251 22644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:45805 every 8 connection(s)
I20260923 18:05:52.401218 22516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:52.407541 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22516
I20260923 18:05:52.407681 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:52.407970 18611 external_mini_cluster.cc:1428] Setting key 52423d95a33605c21c8bf3cdfbfb3a4c
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:52.421125 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.403467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:37145 (local address 127.18.44.254:45043)
0923 18:05:52.403713 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:52.403718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:52.404918 (+ 1200us) server_negotiation.cc:407] Connection header received
0923 18:05:52.406463 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:52.406473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:52.406701 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:52.406808 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:52.408760 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.409458 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.411455 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.411892 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.412016 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.415934 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:52.415938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:52.415940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:52.415972 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:52.417578 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.418672 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.418678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.418680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.418851 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.419702 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.419706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.419707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.419945 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:52.420043 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:52.420851 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 18:05:52.420995 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 18:05:52.422219 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.403938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45043 (local address 127.18.44.195:37145)
0923 18:05:52.404476 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:52.404523 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:52.406119 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:52.407043 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:52.407051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:52.407648 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:52.408560 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:52.408582 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.409663 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.409667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:52.411160 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:52.411177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.412100 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.412104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:52.412396 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.413465 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:52.413485 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:52.415677 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:52.417793 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:52.417800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:52.417821 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:52.418466 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:52.418990 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:52.418993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:52.418995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:52.419560 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:52.420101 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:52.420108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:52.420528 (+ 420us) client_negotiation.cc:769] Sending connection context
0923 18:05:52.420719 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 18:05:52.421116 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
I20260923 18:05:52.423830 22645 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45043
I20260923 18:05:52.424328 22645 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:52.424926 22645 heartbeater.cc:511] Master 127.18.44.254:45043 requested a full tablet report, sending...
I20260923 18:05:52.426504 22182 ts_manager.cc:194] Registered new tserver with Master: 2f8e301322334065aa0e7d90f2c97248 (127.18.44.195:45805)
I20260923 18:05:52.427153 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:37145
I20260923 18:05:52.436698 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:05:52.467546 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.456363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.18.44.193:36585)
0923 18:05:52.456713 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:52.456721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:52.456750 (+ 29us) server_negotiation.cc:407] Connection header received
0923 18:05:52.456932 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:52.456938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:52.457139 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:52.457336 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:52.458127 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.458921 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.459859 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.460275 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.460379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.463172 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:52.463232 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:52.463247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:52.463311 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:52.465454 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.466035 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.466042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.466043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.466141 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.466445 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.466448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.466449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.466730 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:52.466957 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:52.467165 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 18:05:52.467361 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:52.481599 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:52.471495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.18.44.254:45043)
0923 18:05:52.471641 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:52.471644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:52.471681 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:05:52.471955 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:52.471958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:52.472014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:52.472103 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:52.472878 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.473522 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.474492 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:52.474744 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:52.474838 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:52.478028 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:52.478031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:52.478034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:52.478071 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:52.479554 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.480056 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.480059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.480060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.480147 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:52.480438 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:52.480441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:52.480443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:52.480706 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:52.480835 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:52.481102 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 18:05:52.481326 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 18:05:52.482079 22182 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50784
I20260923 18:05:52.483109 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22248
I20260923 18:05:52.489385 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22382
I20260923 18:05:52.495226 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22516
I20260923 18:05:52.500133 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22154
2026-09-23T18:05:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2371 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22657](info): set up 2 sockets
Sep 23 18:05:52 dist-test-slave-rvjj krb5kdc[22657](info): commencing operation
krb5kdc: starting...
W20260923 18:05:53.517611 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:53 dist-test-slave-rvjj krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:53.668692 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33075
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:33075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:53.785843 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:53.786190 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:53.786242 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:53.790688 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:53.790783 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:53.790807 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:53.790822 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:53.790836 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:53.796756 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:33075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:53.797894 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:53.798936 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:53.808260 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:53.808291 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:53.808260 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:53.808323 22670 server_base.cc:1034] running on GCE node
I20260923 18:05:53.809218 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:53.810710 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:53.811947 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790186753811931 us; error 44 us; skew 500 ppm
Sep 23 18:05:53 dist-test-slave-rvjj krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186753, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:53.815196 22670 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:53.816102 22670 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:53.816180 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:53.816408 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:53.818589 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "c5287bb6aeee49f89393962c5940ce48"
format_stamp: "Formatted at 2026-09-23 18:05:53 on dist-test-slave-rvjj"
server_key: "c4cd59f04d61094ebb87c6325f2f6e25"
server_key_iv: "9c993902eaea0cd334ba85630f723217"
server_key_version: "encryptionkey@0"
I20260923 18:05:53.819176 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "c5287bb6aeee49f89393962c5940ce48"
format_stamp: "Formatted at 2026-09-23 18:05:53 on dist-test-slave-rvjj"
server_key: "c4cd59f04d61094ebb87c6325f2f6e25"
server_key_iv: "9c993902eaea0cd334ba85630f723217"
server_key_version: "encryptionkey@0"
I20260923 18:05:53.823427 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:05:53.826316 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:53.828038 22670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:53.828199 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "c5287bb6aeee49f89393962c5940ce48"
format_stamp: "Formatted at 2026-09-23 18:05:53 on dist-test-slave-rvjj"
server_key: "c4cd59f04d61094ebb87c6325f2f6e25"
server_key_iv: "9c993902eaea0cd334ba85630f723217"
server_key_version: "encryptionkey@0"
I20260923 18:05:53.828333 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:53.841477 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:53.845201 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:53.845408 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:53.852651 22670 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:33075
I20260923 18:05:53.852957 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:33075 every 8 connection(s)
I20260923 18:05:53.853384 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:53.856482 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:53.857371 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22670
I20260923 18:05:53.857496 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:53.857743 18611 external_mini_cluster.cc:1428] Setting key eee773da674b236491adec187505440f
I20260923 18:05:53.864064 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: Bootstrap starting.
I20260923 18:05:53.867180 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: Neither blocks nor log segments found. Creating new log.
Sep 23 18:05:53 dist-test-slave-rvjj krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:53.868194 22737 log.cc:824] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:53.871022 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: No bootstrap required, opened a new log
I20260923 18:05:53.874315 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:53.859168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.18.44.254:33075)
0923 18:05:53.859666 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:53.859681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:53.859718 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:05:53.860449 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:53.860467 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:53.860822 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:53.861149 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:53.861990 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:53.863582 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:53.864846 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:53.865356 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:53.865484 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:53.868813 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:53.868830 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:53.868855 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:53.868889 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:53.870985 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:53.871642 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:53.871649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:53.871654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:53.871822 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:53.872155 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:53.872161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:53.872163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:53.872598 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:53.872757 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:53.873053 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 18:05:53.873377 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":153,"threads_started":1}
I20260923 18:05:53.875316 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } }
I20260923 18:05:53.875577 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:53.875624 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5287bb6aeee49f89393962c5940ce48, State: Initialized, Role: FOLLOWER
I20260923 18:05:53.876087 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } }
I20260923 18:05:53.876219 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:53.876282 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:53.876374 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:53.877489 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } }
I20260923 18:05:53.877851 22737 leader_election.cc:302] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5287bb6aeee49f89393962c5940ce48; no voters:
I20260923 18:05:53.878198 22737 leader_election.cc:288] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:53.878351 22742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:53.878669 22742 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [term 1 LEADER]: Becoming Leader. State: Replica: c5287bb6aeee49f89393962c5940ce48, State: Running, Role: LEADER
I20260923 18:05:53.878996 22742 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } }
I20260923 18:05:53.879536 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:53.880779 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5287bb6aeee49f89393962c5940ce48. Latest consensus state: current_term: 1 leader_uuid: "c5287bb6aeee49f89393962c5940ce48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } } }
I20260923 18:05:53.880923 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:53.881335 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5287bb6aeee49f89393962c5940ce48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5287bb6aeee49f89393962c5940ce48" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33075 } } }
I20260923 18:05:53.881445 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:53.881567 22751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:53.884793 22751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:53.890434 22751 catalog_manager.cc:1347] Generated new cluster ID: 7d0770be063746a7a8ca66a1d90b1aa1
I20260923 18:05:53.890525 22751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:53.905463 22751 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:53.906520 22751 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:53.925891 22751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: Generated new TSK 0
I20260923 18:05:53.926920 22751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:53.969178 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:54.097216 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:54.097518 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:54.097586 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:54.102097 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:54.102195 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:54.102283 22763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:54.108289 22763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:54.109510 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:54.110608 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:54.121299 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:54.121300 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:54.121299 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:54.121800 22763 server_base.cc:1034] running on GCE node
I20260923 18:05:54.122226 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:54.122893 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:54.124135 22763 hybrid_clock.cc:648] HybridClock initialized: now 1790186754124082 us; error 68 us; skew 500 ppm
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:54.127384 22763 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:54.128237 22763 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:54.128315 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:54.128501 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:54.130932 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "3886420afc274c1e9e223330fefcc7b6"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "b2e94a58e26d4ed094d10e48f526a833"
server_key_iv: "a0b844438dafda346ebc89b84d5bdc35"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.131564 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "3886420afc274c1e9e223330fefcc7b6"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "b2e94a58e26d4ed094d10e48f526a833"
server_key_iv: "a0b844438dafda346ebc89b84d5bdc35"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.136061 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:54.138849 22777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.140460 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:05:54.140602 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "3886420afc274c1e9e223330fefcc7b6"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "b2e94a58e26d4ed094d10e48f526a833"
server_key_iv: "a0b844438dafda346ebc89b84d5bdc35"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.140723 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:54.166590 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:54.170540 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:54.170768 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:54.171454 22763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:54.172405 22763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:54.172458 22763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.172504 22763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:54.172528 22763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.182662 22763 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:34079
I20260923 18:05:54.182679 22890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:34079 every 8 connection(s)
I20260923 18:05:54.183396 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:54.187628 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22763
I20260923 18:05:54.187768 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:54.188095 18611 external_mini_cluster.cc:1428] Setting key 98c36072c84764fabefb2462df0c8219
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:54.200182 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.185395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:43349 (local address 127.18.44.254:33075)
0923 18:05:54.185598 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.185602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.186502 (+ 900us) server_negotiation.cc:407] Connection header received
0923 18:05:54.188167 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.188171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.188306 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.188391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:54.190165 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.190723 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.192368 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.192724 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.192845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.195881 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:54.195885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:54.195887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:54.195915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:54.197476 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.198215 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.198218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.198220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.198314 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.198824 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.198827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.198828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.199038 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:54.199121 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.199863 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.199994 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 18:05:54.200915 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.185789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33075 (local address 127.18.44.193:43349)
0923 18:05:54.186308 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.186363 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.187833 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.188625 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.188633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.189116 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:54.189939 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.189965 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.190895 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.190899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.192225 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.192238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.192921 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.192925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.193089 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.193802 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:54.193821 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:54.195620 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:54.197620 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.197628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.197650 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.198061 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.198440 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.198443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.198444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.198607 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.199197 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:54.199202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:54.199525 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.199699 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.199994 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:54.202199 22891 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33075
I20260923 18:05:54.202448 22891 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:54.202905 22891 heartbeater.cc:511] Master 127.18.44.254:33075 requested a full tablet report, sending...
I20260923 18:05:54.204530 22700 ts_manager.cc:194] Registered new tserver with Master: 3886420afc274c1e9e223330fefcc7b6 (127.18.44.193:34079)
I20260923 18:05:54.205673 22700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:43349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:54.219214 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:54.335713 22896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:54.336011 22896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:54.336057 22896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:54.340333 22896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:54.340416 22896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:54.340481 22896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:54.346230 22896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:54.347417 22896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:54.348493 22896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:54.358480 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:54.358506 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:54.358553 22896 server_base.cc:1034] running on GCE node
W20260923 18:05:54.358502 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:54.359385 22896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:54.360081 22896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:54.361332 22896 hybrid_clock.cc:648] HybridClock initialized: now 1790186754361302 us; error 50 us; skew 500 ppm
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:54.370554 22896 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:05:54.371524 22896 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:54.371603 22896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:54.371825 22896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:54.373953 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "20781c0733104a48882a2148afe4ca4e"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "481204cb3332b6f66e1a4afabb09389a"
server_key_iv: "0e631ed8cd4cf2d639840736dd4d5ae1"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.374498 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "20781c0733104a48882a2148afe4ca4e"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "481204cb3332b6f66e1a4afabb09389a"
server_key_iv: "0e631ed8cd4cf2d639840736dd4d5ae1"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.378436 22896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:05:54.381162 22910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.382530 22896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:05:54.382649 22896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "20781c0733104a48882a2148afe4ca4e"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "481204cb3332b6f66e1a4afabb09389a"
server_key_iv: "0e631ed8cd4cf2d639840736dd4d5ae1"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.382853 22896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:54.407609 22896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:54.411249 22896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:54.411552 22896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:54.412209 22896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:54.413103 22896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:54.413153 22896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.413209 22896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:54.413239 22896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.424049 22896 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39931
I20260923 18:05:54.424119 23023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39931 every 8 connection(s)
I20260923 18:05:54.424826 22896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:54.426625 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 22896
I20260923 18:05:54.426837 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:54.427169 18611 external_mini_cluster.cc:1428] Setting key 62382ee119189cdc443060d0912312b0
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:05:54.442896 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.426866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:54235 (local address 127.18.44.254:33075)
0923 18:05:54.427034 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.427039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.428662 (+ 1623us) server_negotiation.cc:407] Connection header received
0923 18:05:54.430405 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.430409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.430481 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.430601 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:54.432042 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.432847 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.434955 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.435205 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.435389 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.438380 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:54.438383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:54.438385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:54.438414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:54.440232 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.440866 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.440870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.440872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.440961 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.441385 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.441389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.441391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.441669 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:54.441777 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.442527 (+ 750us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.442708 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:05:54.443873 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.427365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33075 (local address 127.18.44.194:54235)
0923 18:05:54.428488 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.428540 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.430097 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.430795 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.430806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.431226 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:54.431870 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.431888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.433216 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.433219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.434749 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.434767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.435468 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.435473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.435721 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.436417 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:54.436441 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:54.438152 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:54.440385 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.440393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.440423 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.440742 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.441094 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.441097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.441099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.441281 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.441792 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:54.441800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:54.442124 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.442442 (+ 318us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.442853 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":179,"threads_started":1}
I20260923 18:05:54.445379 23024 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33075
I20260923 18:05:54.445655 23024 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:54.446168 23024 heartbeater.cc:511] Master 127.18.44.254:33075 requested a full tablet report, sending...
I20260923 18:05:54.447443 22700 ts_manager.cc:194] Registered new tserver with Master: 20781c0733104a48882a2148afe4ca4e (127.18.44.194:39931)
I20260923 18:05:54.448119 22700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:54235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:05:54.463375 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:54.597847 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:54.598205 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:54.598254 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:54.602751 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:54.602944 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:54.603068 23029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:54.609483 23029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:54.610962 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:54.612812 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:54.624078 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:54.624119 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:54.624073 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:54.624795 23029 server_base.cc:1034] running on GCE node
I20260923 18:05:54.625371 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:54.626107 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:54.627358 23029 hybrid_clock.cc:648] HybridClock initialized: now 1790186754627334 us; error 34 us; skew 500 ppm
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:54.630530 23029 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:05:54.631456 23029 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:54.631546 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:54.631870 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:54.634399 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "9ada7037cc2344b59a90c539277c10eb"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "0c19933048cfb5f71206e6edcf3cac21"
server_key_iv: "dd4dc0b3e08a42bfc1939ba7d874a353"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.635031 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "9ada7037cc2344b59a90c539277c10eb"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "0c19933048cfb5f71206e6edcf3cac21"
server_key_iv: "dd4dc0b3e08a42bfc1939ba7d874a353"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.639442 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:54.642818 23043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.644644 23029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:54.644824 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "9ada7037cc2344b59a90c539277c10eb"
format_stamp: "Formatted at 2026-09-23 18:05:54 on dist-test-slave-rvjj"
server_key: "0c19933048cfb5f71206e6edcf3cac21"
server_key_iv: "dd4dc0b3e08a42bfc1939ba7d874a353"
server_key_version: "encryptionkey@0"
I20260923 18:05:54.644961 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:54.667160 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:54.671046 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:54.671305 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:54.671950 23029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:54.672861 23029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:54.672912 23029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.672966 23029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:54.672995 23029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:54.683089 23029 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40553
I20260923 18:05:54.683110 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40553 every 8 connection(s)
I20260923 18:05:54.683820 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:54.691891 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23029
I20260923 18:05:54.692056 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:54.692355 18611 external_mini_cluster.cc:1428] Setting key 2633b91a62e59fdd382cccc7e516860b
Sep 23 18:05:54 dist-test-slave-rvjj krb5kdc[22657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186754, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:54.699347 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.685786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:41033 (local address 127.18.44.254:33075)
0923 18:05:54.685958 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.685963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.686956 (+ 993us) server_negotiation.cc:407] Connection header received
0923 18:05:54.688471 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.688475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.688546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.688642 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:54.690012 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.690584 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.692088 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.692372 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.692489 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.695571 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:54.695574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:54.695576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:54.695603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:54.696853 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.697416 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.697419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.697420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.697502 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.697965 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.697968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.697969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.698233 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:54.698320 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.698953 (+ 633us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.699145 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:05:54.700160 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.686159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33075 (local address 127.18.44.195:41033)
0923 18:05:54.686779 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.686832 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.688218 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.688846 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.688854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.689294 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:54.689830 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.689846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.690751 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.690754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.691904 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.691920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.692592 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.692597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.692808 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.693500 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:54.693524 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:54.695406 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:54.696975 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.696980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.696996 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.697283 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.697584 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:54.697587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:54.697588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:54.697738 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:54.698343 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:54.698349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:54.698630 (+ 281us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.698867 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.699170 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":149,"threads_started":1}
I20260923 18:05:54.701535 23157 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33075
I20260923 18:05:54.701771 23157 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:54.702224 23157 heartbeater.cc:511] Master 127.18.44.254:33075 requested a full tablet report, sending...
I20260923 18:05:54.703377 22700 ts_manager.cc:194] Registered new tserver with Master: 9ada7037cc2344b59a90c539277c10eb (127.18.44.195:40553)
I20260923 18:05:54.703966 22700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:41033
I20260923 18:05:54.706822 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:05:54.718582 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.708949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.18.44.254:33075)
0923 18:05:54.709077 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.709080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.709254 (+ 174us) server_negotiation.cc:407] Connection header received
0923 18:05:54.709332 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.709335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.709379 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.709454 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:54.710297 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.710906 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.711897 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.712122 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.712221 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.714144 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:54.714158 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:54.714163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:54.714202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:54.716410 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.716980 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.716984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.716986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.717076 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:54.717364 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:54.717367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:54.717369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:54.717705 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:54.717993 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.718265 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.718404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:05:54.722599 22700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:54.725962 22700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:54.743155 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.736352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.18.44.193:34079)
0923 18:05:54.737023 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.737031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.737376 (+ 345us) server_negotiation.cc:407] Connection header received
0923 18:05:54.737609 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.737615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.737854 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.737984 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:54.738407 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739605 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.741443 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.742235 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.742357 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.742513 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.742878 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.742991 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":131,"threads_started":1}
I20260923 18:05:54.743167 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.736320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34079 (local address 127.0.0.1:46932)
0923 18:05:54.737220 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.737279 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.737450 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.738172 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.738175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.738191 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:54.738289 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.738294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739718 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.739720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.741309 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.741320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.742397 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.742401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.742622 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.742676 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.742803 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.742993 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":226,"thread_start_us":120,"threads_started":1}
I20260923 18:05:54.743167 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.736502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39931 (local address 127.0.0.1:50166)
0923 18:05:54.737238 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.737280 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.737418 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.738083 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.738089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.738113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:54.738238 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.738244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739843 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.739847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.741210 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.741219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.742397 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.742401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.742615 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.742678 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.742803 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.742992 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":113,"thread_start_us":87,"threads_started":1}
I20260923 18:05:54.744130 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.736325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40553 (local address 127.0.0.1:58890)
0923 18:05:54.737880 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.737893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.737987 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.738562 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.738565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.738578 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:54.738654 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.738658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739678 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.739681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.741320 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.741327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.743741 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.743745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.743833 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.743855 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.743922 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.744000 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"mutex_wait_us":515,"thread_start_us":83,"threads_started":1}
I20260923 18:05:54.744470 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.737191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.18.44.194:39931)
0923 18:05:54.737632 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.737647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.737668 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:05:54.737739 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.737743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.737896 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.738026 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:54.738358 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739615 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.741336 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.742235 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.742353 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.742513 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.744154 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.744273 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":151,"threads_started":1}
I20260923 18:05:54.744599 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.736563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.18.44.195:40553)
0923 18:05:54.737711 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.737725 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.737990 (+ 265us) server_negotiation.cc:407] Connection header received
0923 18:05:54.738145 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.738150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.738319 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.738442 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:54.738769 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.739546 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.742590 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.743416 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.743523 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.743664 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.744308 (+ 644us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.744411 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":102,"threads_started":1}
I20260923 18:05:54.746686 22825 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd74dae1f848dc82832700f37fb187 (DEFAULT_TABLE table=test-table [id=1e7f5b978d9f46258438dac11f31b2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:54.747045 22958 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd74dae1f848dc82832700f37fb187 (DEFAULT_TABLE table=test-table [id=1e7f5b978d9f46258438dac11f31b2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:54.748023 22958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fd74dae1f848dc82832700f37fb187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:54.748014 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fd74dae1f848dc82832700f37fb187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:54.749715 23091 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd74dae1f848dc82832700f37fb187 (DEFAULT_TABLE table=test-table [id=1e7f5b978d9f46258438dac11f31b2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:54.750710 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fd74dae1f848dc82832700f37fb187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:54.755873 23174 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: Bootstrap starting.
I20260923 18:05:54.757405 23175 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Bootstrap starting.
I20260923 18:05:54.757406 23176 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Bootstrap starting.
I20260923 18:05:54.758934 23174 tablet_bootstrap.cc:654] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:54.759158 23175 tablet_bootstrap.cc:654] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:54.759433 23176 tablet_bootstrap.cc:654] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:54.759938 23174 log.cc:824] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:54.760087 23175 log.cc:824] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:54.760551 23176 log.cc:824] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:54.762264 23175 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: No bootstrap required, opened a new log
I20260923 18:05:54.762311 23174 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: No bootstrap required, opened a new log
I20260923 18:05:54.762311 23176 tablet_bootstrap.cc:492] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: No bootstrap required, opened a new log
I20260923 18:05:54.762475 23175 ts_tablet_manager.cc:1397] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 18:05:54.762490 23174 ts_tablet_manager.cc:1397] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 18:05:54.762490 23176 ts_tablet_manager.cc:1397] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:54.765522 23175 raft_consensus.cc:348] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.765802 23175 raft_consensus.cc:374] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:54.765887 23175 raft_consensus.cc:729] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20781c0733104a48882a2148afe4ca4e, State: Initialized, Role: FOLLOWER
I20260923 18:05:54.766340 23175 consensus_queue.cc:260] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.766335 23176 raft_consensus.cc:348] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.766675 23176 raft_consensus.cc:374] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:54.766788 23176 raft_consensus.cc:729] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ada7037cc2344b59a90c539277c10eb, State: Initialized, Role: FOLLOWER
I20260923 18:05:54.767053 23024 heartbeater.cc:503] Master 127.18.44.254:33075 was elected leader, sending a full tablet report...
I20260923 18:05:54.766945 23174 raft_consensus.cc:348] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.767198 23174 raft_consensus.cc:374] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:54.767246 23174 raft_consensus.cc:729] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3886420afc274c1e9e223330fefcc7b6, State: Initialized, Role: FOLLOWER
I20260923 18:05:54.767340 23176 consensus_queue.cc:260] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.767472 23175 ts_tablet_manager.cc:1428] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:05:54.767732 23174 consensus_queue.cc:260] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.768504 23176 ts_tablet_manager.cc:1428] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:05:54.768837 23174 ts_tablet_manager.cc:1428] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260923 18:05:54.768940 23157 heartbeater.cc:503] Master 127.18.44.254:33075 was elected leader, sending a full tablet report...
I20260923 18:05:54.769191 22891 heartbeater.cc:503] Master 127.18.44.254:33075 was elected leader, sending a full tablet report...
I20260923 18:05:54.823407 23182 raft_consensus.cc:482] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:54.823668 23182 raft_consensus.cc:504] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.824946 23182 leader_election.cc:288] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ada7037cc2344b59a90c539277c10eb (127.18.44.195:40553), 20781c0733104a48882a2148afe4ca4e (127.18.44.194:39931)
I20260923 18:05:54.831956 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.825595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:48475 (local address 127.18.44.195:40553)
0923 18:05:54.825737 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.825741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.826025 (+ 284us) server_negotiation.cc:407] Connection header received
0923 18:05:54.826130 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.826134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.826210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.826339 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:54.826929 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.827856 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.830028 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.830829 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.831094 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.831147 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.831654 (+ 507us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.831778 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:05:54.831956 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.825467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:42139 (local address 127.18.44.194:39931)
0923 18:05:54.825830 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.825837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.825928 (+ 91us) server_negotiation.cc:407] Connection header received
0923 18:05:54.826154 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.826157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.826210 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.826339 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:05:54.826907 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.827856 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.830029 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.830830 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.831060 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.831136 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.831654 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.831778 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 18:05:54.831959 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.825453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40553 (local address 127.18.44.193:48475)
0923 18:05:54.825913 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.825926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.826023 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.826423 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.826427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.826480 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:54.826796 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.826804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.828098 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.828103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.829759 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.829783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.831118 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.831123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.831343 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.831399 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.831530 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.831675 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":125,"spinlock_wait_cycles":12672,"thread_start_us":185,"threads_started":1}
I20260923 18:05:54.831995 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.825356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39931 (local address 127.18.44.193:42139)
0923 18:05:54.825819 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.825835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:54.825988 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:54.826423 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:54.826427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:54.826479 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:05:54.826725 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.826735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.828098 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.828103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.829759 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:54.829782 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.831118 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.831123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:54.831358 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.831399 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 18:05:54.831530 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 18:05:54.831720 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":293,"spinlock_wait_cycles":32256,"thread_start_us":147,"threads_started":1}
I20260923 18:05:54.833261 22978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd74dae1f848dc82832700f37fb187" candidate_uuid: "3886420afc274c1e9e223330fefcc7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20781c0733104a48882a2148afe4ca4e" is_pre_election: true
I20260923 18:05:54.833395 23110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd74dae1f848dc82832700f37fb187" candidate_uuid: "3886420afc274c1e9e223330fefcc7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ada7037cc2344b59a90c539277c10eb" is_pre_election: true
I20260923 18:05:54.833724 23110 raft_consensus.cc:2454] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3886420afc274c1e9e223330fefcc7b6 in term 0.
I20260923 18:05:54.833724 22978 raft_consensus.cc:2454] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3886420afc274c1e9e223330fefcc7b6 in term 0.
I20260923 18:05:54.834470 22779 leader_election.cc:302] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3886420afc274c1e9e223330fefcc7b6, 9ada7037cc2344b59a90c539277c10eb; no voters:
I20260923 18:05:54.834874 23182 raft_consensus.cc:2781] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:54.835098 23182 raft_consensus.cc:482] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:54.835160 23182 raft_consensus.cc:3037] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:54.836727 23182 raft_consensus.cc:504] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.837311 23182 leader_election.cc:288] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9ada7037cc2344b59a90c539277c10eb (127.18.44.195:40553), 20781c0733104a48882a2148afe4ca4e (127.18.44.194:39931)
I20260923 18:05:54.838233 22978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd74dae1f848dc82832700f37fb187" candidate_uuid: "3886420afc274c1e9e223330fefcc7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20781c0733104a48882a2148afe4ca4e"
I20260923 18:05:54.838323 23110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd74dae1f848dc82832700f37fb187" candidate_uuid: "3886420afc274c1e9e223330fefcc7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ada7037cc2344b59a90c539277c10eb"
I20260923 18:05:54.838450 23110 raft_consensus.cc:3037] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:54.838450 22978 raft_consensus.cc:3037] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:54.839977 22978 raft_consensus.cc:2454] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3886420afc274c1e9e223330fefcc7b6 in term 1.
I20260923 18:05:54.839977 23110 raft_consensus.cc:2454] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3886420afc274c1e9e223330fefcc7b6 in term 1.
I20260923 18:05:54.840492 22779 leader_election.cc:302] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3886420afc274c1e9e223330fefcc7b6, 9ada7037cc2344b59a90c539277c10eb; no voters:
I20260923 18:05:54.840770 23182 raft_consensus.cc:2781] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:54.841073 23182 raft_consensus.cc:686] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3886420afc274c1e9e223330fefcc7b6, State: Running, Role: LEADER
I20260923 18:05:54.841521 23182 consensus_queue.cc:237] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } }
I20260923 18:05:54.846477 22701 catalog_manager.cc:5495] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3886420afc274c1e9e223330fefcc7b6 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "3886420afc274c1e9e223330fefcc7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3886420afc274c1e9e223330fefcc7b6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:54.878324 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.873344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.18.44.193:34079)
0923 18:05:54.873615 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:54.873620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:54.873640 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:05:54.873760 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:54.873763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:54.873841 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:54.873969 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:05:54.874382 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.875081 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.876193 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:54.876474 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:54.876598 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:54.877114 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:05:54.877759 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:05:54.877903 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:54.878035 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 18:05:54.878124 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:05:54.886726 22978 raft_consensus.cc:1261] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 1 FOLLOWER]: Refusing update from remote peer 3886420afc274c1e9e223330fefcc7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:54.886843 23110 raft_consensus.cc:1261] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 1 FOLLOWER]: Refusing update from remote peer 3886420afc274c1e9e223330fefcc7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:54.888032 23182 consensus_queue.cc:1035] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ada7037cc2344b59a90c539277c10eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:54.888331 23185 consensus_queue.cc:1035] T d6fd74dae1f848dc82832700f37fb187 P 3886420afc274c1e9e223330fefcc7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20781c0733104a48882a2148afe4ca4e" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:54.900919 23189 mvcc.cc:205] Tried to move back new op lower bound from 7332604948006125568 to 7332604947835142144. Current Snapshot: MvccSnapshot[applied={T|T < 7332604948006125568}]
I20260923 18:05:54.915139 22700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:54.915462 22700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:54.919154 22700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5287bb6aeee49f89393962c5940ce48: Sending DeleteTablet for 3 replicas of tablet d6fd74dae1f848dc82832700f37fb187
I20260923 18:05:54.920413 22825 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd74dae1f848dc82832700f37fb187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:54 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:46932
I20260923 18:05:54.920842 22958 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd74dae1f848dc82832700f37fb187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:54 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:50166
I20260923 18:05:54.921083 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd74dae1f848dc82832700f37fb187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:54 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:58890
I20260923 18:05:54.921512 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22763
I20260923 18:05:54.921681 23199 tablet_replica.cc:330] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: stopping tablet replica
I20260923 18:05:54.922940 23199 raft_consensus.cc:2229] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:54.923488 23199 raft_consensus.cc:2258] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:54.924201 23200 tablet_replica.cc:330] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: stopping tablet replica
I20260923 18:05:54.924705 23200 raft_consensus.cc:2229] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:54.925112 23200 raft_consensus.cc:2258] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:54.925578 23199 ts_tablet_manager.cc:1905] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:54.927173 23200 ts_tablet_manager.cc:1905] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:54.928647 23199 ts_tablet_manager.cc:1918] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:54.928779 23199 log.cc:1196] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/d6fd74dae1f848dc82832700f37fb187
I20260923 18:05:54.929417 23199 ts_tablet_manager.cc:1939] T d6fd74dae1f848dc82832700f37fb187 P 20781c0733104a48882a2148afe4ca4e: Deleting consensus metadata
W20260923 18:05:54.929869 22685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:34079 (error 108)
I20260923 18:05:54.930380 23200 ts_tablet_manager.cc:1918] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:54.930706 23200 log.cc:1196] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/d6fd74dae1f848dc82832700f37fb187
I20260923 18:05:54.930869 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22896
I20260923 18:05:54.931830 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.930988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34079 (local address 127.0.0.1:46946)
0923 18:05:54.931614 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.931721 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496}
W20260923 18:05:54.932085 22685 catalog_manager.cc:4584] TS 3886420afc274c1e9e223330fefcc7b6 (127.18.44.193:34079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6fd74dae1f848dc82832700f37fb187: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34079: connect: Connection refused (error 111)
I20260923 18:05:54.933285 23200 ts_tablet_manager.cc:1939] T d6fd74dae1f848dc82832700f37fb187 P 9ada7037cc2344b59a90c539277c10eb: Deleting consensus metadata
I20260923 18:05:54.935019 22686 catalog_manager.cc:4841] TS 9ada7037cc2344b59a90c539277c10eb (127.18.44.195:40553): tablet d6fd74dae1f848dc82832700f37fb187 (table test-table [id=1e7f5b978d9f46258438dac11f31b2df]) successfully deleted
I20260923 18:05:54.937430 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23029
I20260923 18:05:54.939493 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:54.939002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39931 (local address 127.0.0.1:50170)
0923 18:05:54.939219 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:54.939344 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:39931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 18:05:54.943101 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 22670
2026-09-23T18:05:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2443 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T18:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:54Z Disabled control of system clock
I20260923 18:05:54.977238 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37315
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 18:05:55.109234 23205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:55.109577 23205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:55.109617 23205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:55.115473 23205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:55.115569 23205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:55.115585 23205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:55.115603 23205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:55.115643 23205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:55.122133 23205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:55.123453 23205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:55.124980 23205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:55.135390 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.135408 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.135390 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.135696 23205 server_base.cc:1034] running on GCE node
I20260923 18:05:55.136683 23205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:55.138304 23205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:55.139557 23205 hybrid_clock.cc:648] HybridClock initialized: now 1790186755139556 us; error 88 us; skew 500 ppm
I20260923 18:05:55.142584 23205 webserver.cc:460] Webserver started at http://127.18.44.254:45361/ using document root <none> and password file <none>
I20260923 18:05:55.143550 23205 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:55.143654 23205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:55.143918 23205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:55.147238 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "d51e4cc402fa4478a8dfdf993b738cbd"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "83187a65404262e6c30e1cf181792151"
server_key_iv: "76d85cf0376405a352482511d693bdac"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.148301 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "d51e4cc402fa4478a8dfdf993b738cbd"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "83187a65404262e6c30e1cf181792151"
server_key_iv: "76d85cf0376405a352482511d693bdac"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.154640 23205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 18:05:55.158852 23219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.160876 23205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 18:05:55.161079 23205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "d51e4cc402fa4478a8dfdf993b738cbd"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "83187a65404262e6c30e1cf181792151"
server_key_iv: "76d85cf0376405a352482511d693bdac"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.161381 23205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:55.168555 23205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:55.169445 23205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:55.169653 23205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:55.179527 23205 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37315
I20260923 18:05:55.179607 23271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37315 every 8 connection(s)
I20260923 18:05:55.181003 23205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:55.185650 23272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:55.185962 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23205
I20260923 18:05:55.186158 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:55.186546 18611 external_mini_cluster.cc:1428] Setting key a932504f6a6848cce92436dbab530b7b
I20260923 18:05:55.193465 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: Bootstrap starting.
I20260923 18:05:55.194943 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.188647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.18.44.254:37315)
0923 18:05:55.189801 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:55.189827 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:55.189889 (+ 62us) server_negotiation.cc:407] Connection header received
0923 18:05:55.190915 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:55.190943 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:55.191620 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:55.192060 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:55.193144 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:55.193151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:55.193167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:55.193179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:55.193261 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:55.193281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:55.193443 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:55.193519 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 18:05:55.193931 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":180,"threads_started":1}
I20260923 18:05:55.196221 23272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:55.197190 23272 log.cc:824] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:55.200169 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: No bootstrap required, opened a new log
I20260923 18:05:55.204850 23272 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } }
I20260923 18:05:55.205221 23272 raft_consensus.cc:374] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:55.205282 23272 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51e4cc402fa4478a8dfdf993b738cbd, State: Initialized, Role: FOLLOWER
I20260923 18:05:55.205754 23272 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } }
I20260923 18:05:55.205899 23272 raft_consensus.cc:388] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:55.205942 23272 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:55.206013 23272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:55.207007 23272 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } }
I20260923 18:05:55.207407 23272 leader_election.cc:302] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d51e4cc402fa4478a8dfdf993b738cbd; no voters:
I20260923 18:05:55.207756 23272 leader_election.cc:288] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:55.208120 23277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:55.208477 23277 raft_consensus.cc:686] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [term 1 LEADER]: Becoming Leader. State: Replica: d51e4cc402fa4478a8dfdf993b738cbd, State: Running, Role: LEADER
I20260923 18:05:55.208943 23277 consensus_queue.cc:237] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } }
I20260923 18:05:55.209228 23272 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:55.211385 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51e4cc402fa4478a8dfdf993b738cbd. Latest consensus state: current_term: 1 leader_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } } }
I20260923 18:05:55.211611 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:55.211434 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e4cc402fa4478a8dfdf993b738cbd" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37315 } } }
I20260923 18:05:55.211730 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:55.215318 23291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:55.215440 23291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:55.215521 23292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:55.216260 23292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:55.225999 23292 catalog_manager.cc:1347] Generated new cluster ID: ba443ea2f4d14fdd99ae3376290ca1be
I20260923 18:05:55.226269 23292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:55.238456 23292 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:55.238827 23292 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:55.252813 23292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: Generated new TSK 0
I20260923 18:05:55.253635 23292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:05:55.258545 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--builtin_ntp_servers=127.18.44.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 18:05:55.401892 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:55.402235 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:55.402279 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:55.408659 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:55.408852 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:55.408932 23296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:55.417766 23296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:55.419211 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:55.421155 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:55.443437 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.443455 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.443859 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.445098 23296 server_base.cc:1034] running on GCE node
I20260923 18:05:55.445820 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:55.446933 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:55.450445 23296 hybrid_clock.cc:648] HybridClock initialized: now 1790186755450139 us; error 323 us; skew 500 ppm
I20260923 18:05:55.453225 23296 webserver.cc:460] Webserver started at http://127.18.44.193:43713/ using document root <none> and password file <none>
I20260923 18:05:55.454262 23296 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:55.454386 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:55.454631 23296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:55.457345 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "a74b3d6143174fd8b8e9fb2a329eab98"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "63d50fb99a0ae1864bda49ebdf6eafb8"
server_key_iv: "0769bd12aa5f9d71a7faf3cb25f4e46e"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.458050 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "a74b3d6143174fd8b8e9fb2a329eab98"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "63d50fb99a0ae1864bda49ebdf6eafb8"
server_key_iv: "0769bd12aa5f9d71a7faf3cb25f4e46e"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.463881 23296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:05:55.468334 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.470405 23296 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:05:55.470762 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "a74b3d6143174fd8b8e9fb2a329eab98"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "63d50fb99a0ae1864bda49ebdf6eafb8"
server_key_iv: "0769bd12aa5f9d71a7faf3cb25f4e46e"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.470932 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:55.477320 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:55.478096 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:55.478271 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:55.478883 23296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:55.480031 23296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:55.480089 23296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.480134 23296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:55.480149 23296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.492908 23296 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:34535
I20260923 18:05:55.493044 23423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:34535 every 8 connection(s)
I20260923 18:05:55.494055 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:55.499657 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23296
I20260923 18:05:55.499814 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:55.500164 18611 external_mini_cluster.cc:1428] Setting key 49ff2593b020cbac61f063c1f5448592
I20260923 18:05:55.503475 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--builtin_ntp_servers=127.18.44.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 18:05:55.504266 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.497301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35931 (local address 127.18.44.254:37315)
0923 18:05:55.497493 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:55.497497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:55.499500 (+ 2003us) server_negotiation.cc:407] Connection header received
0923 18:05:55.501359 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:55.501364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:55.501441 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:55.501584 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:55.503612 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:55.503617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:55.503621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:55.503645 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:55.503688 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:55.503702 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:55.503852 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:55.504056 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 18:05:55.504117 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 18:05:55.505528 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.497849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37315 (local address 127.18.44.193:35931)
0923 18:05:55.499080 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:55.499155 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:55.500955 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:55.502037 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:55.502049 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:55.502282 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:55.503024 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:55.503026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:55.503091 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:55.503219 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:55.503854 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:55.503865 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:55.503872 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:05:55.504109 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 18:05:55.504454 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":196,"threads_started":1}
I20260923 18:05:55.507315 23424 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37315
I20260923 18:05:55.507783 23424 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:55.508536 23424 heartbeater.cc:511] Master 127.18.44.254:37315 requested a full tablet report, sending...
I20260923 18:05:55.511394 23235 ts_manager.cc:194] Registered new tserver with Master: a74b3d6143174fd8b8e9fb2a329eab98 (127.18.44.193:34535)
W20260923 18:05:55.663837 23427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:55.664565 23427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:55.664644 23427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:55.671226 23427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:55.671428 23427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:55.671510 23427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:05:55.679531 23427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:55.681396 23427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:55.683880 23427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:55.700129 23433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.700155 23432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.700289 23427 server_base.cc:1034] running on GCE node
W20260923 18:05:55.700155 23435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.701363 23427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:55.702196 23427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:55.705643 23427 hybrid_clock.cc:648] HybridClock initialized: now 1790186755705599 us; error 66 us; skew 500 ppm
I20260923 18:05:55.708839 23427 webserver.cc:460] Webserver started at http://127.18.44.194:33285/ using document root <none> and password file <none>
I20260923 18:05:55.709805 23427 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:55.709920 23427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:55.710322 23427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:55.713500 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "a758afbd1edc401d89fb1dcc37543a1b"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "faa0311dc4f5a32a4079490df6781d74"
server_key_iv: "78946f78fa7f751d9f20c71d74b1bbde"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.714222 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "a758afbd1edc401d89fb1dcc37543a1b"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "faa0311dc4f5a32a4079490df6781d74"
server_key_iv: "78946f78fa7f751d9f20c71d74b1bbde"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.720353 23427 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:05:55.725142 23441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.727794 23427 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:05:55.727972 23427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "a758afbd1edc401d89fb1dcc37543a1b"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "faa0311dc4f5a32a4079490df6781d74"
server_key_iv: "78946f78fa7f751d9f20c71d74b1bbde"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.728116 23427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:55.733719 23427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:55.734465 23427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:55.734637 23427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:55.735237 23427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:55.736510 23427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:55.736586 23427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.736653 23427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:55.736685 23427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.747454 23427 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:33505
I20260923 18:05:55.747715 23554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:33505 every 8 connection(s)
I20260923 18:05:55.748548 23427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:05:55.754562 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23427
I20260923 18:05:55.754702 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:05:55.755043 18611 external_mini_cluster.cc:1428] Setting key d08a1b37eedf89006a536327dc52375e
I20260923 18:05:55.757658 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.751151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:51749 (local address 127.18.44.254:37315)
0923 18:05:55.751417 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:55.751430 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:55.753490 (+ 2060us) server_negotiation.cc:407] Connection header received
0923 18:05:55.755172 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:55.755178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:55.755291 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:55.755439 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:55.757068 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:55.757076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:55.757079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:55.757095 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:55.757121 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:55.757189 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:55.757280 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:55.757433 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 18:05:55.757487 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 18:05:55.757839 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--builtin_ntp_servers=127.18.44.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 18:05:55.758797 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.751824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37315 (local address 127.18.44.194:51749)
0923 18:05:55.753196 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:55.753269 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:55.754836 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:55.755691 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:55.755700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:55.755888 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:55.756705 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:55.756708 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:55.756785 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:55.756899 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:55.757280 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:55.757290 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:55.757297 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:05:55.757486 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 18:05:55.757798 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":208,"threads_started":1}
I20260923 18:05:55.760174 23555 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37315
I20260923 18:05:55.760643 23555 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:55.761371 23555 heartbeater.cc:511] Master 127.18.44.254:37315 requested a full tablet report, sending...
I20260923 18:05:55.763021 23235 ts_manager.cc:194] Registered new tserver with Master: a758afbd1edc401d89fb1dcc37543a1b (127.18.44.194:33505)
W20260923 18:05:55.908377 23558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:55.908730 23558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:55.908799 23558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:55.913954 23558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:55.914062 23558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:55.914153 23558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:05:55.919906 23558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:55.921252 23558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:55.923134 23558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:55.936712 23563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.936904 23558 server_base.cc:1034] running on GCE node
W20260923 18:05:55.936753 23566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:55.936724 23564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:55.937716 23558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:55.938444 23558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:55.939741 23558 hybrid_clock.cc:648] HybridClock initialized: now 1790186755939707 us; error 66 us; skew 500 ppm
I20260923 18:05:55.942225 23558 webserver.cc:460] Webserver started at http://127.18.44.195:41687/ using document root <none> and password file <none>
I20260923 18:05:55.942909 23558 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:55.942991 23558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:55.943212 23558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:55.945741 23558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "3b60e0c460324bd39ef0d1d28a10c810"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "be080ee9f14c27c06a0754ffb2d5cd08"
server_key_iv: "f5a008d10efb67ceac557f1d3d30bf46"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.946427 23558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "3b60e0c460324bd39ef0d1d28a10c810"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "be080ee9f14c27c06a0754ffb2d5cd08"
server_key_iv: "f5a008d10efb67ceac557f1d3d30bf46"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.951433 23558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:05:55.955533 23572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.957069 23558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:55.957230 23558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "3b60e0c460324bd39ef0d1d28a10c810"
format_stamp: "Formatted at 2026-09-23 18:05:55 on dist-test-slave-rvjj"
server_key: "be080ee9f14c27c06a0754ffb2d5cd08"
server_key_iv: "f5a008d10efb67ceac557f1d3d30bf46"
server_key_version: "encryptionkey@0"
I20260923 18:05:55.957386 23558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:55.962499 23558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:55.963243 23558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:55.963495 23558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:55.964103 23558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:55.965149 23558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:55.965206 23558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.965248 23558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:55.965265 23558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:55.976279 23558 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40265
I20260923 18:05:55.976346 23685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40265 every 8 connection(s)
I20260923 18:05:55.977319 23558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:05:55.977852 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23558
I20260923 18:05:55.978003 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:05:55.978428 18611 external_mini_cluster.cc:1428] Setting key 942224c3db660dea402d7ed598ffe722
I20260923 18:05:55.986630 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.980416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:44797 (local address 127.18.44.254:37315)
0923 18:05:55.980589 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:55.980594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:55.982481 (+ 1887us) server_negotiation.cc:407] Connection header received
0923 18:05:55.984191 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:55.984195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:55.984264 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:55.984380 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:55.986060 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:55.986063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:55.986065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:55.986079 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:55.986103 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:55.986112 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:55.986215 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:55.986451 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 18:05:55.986499 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 18:05:55.988147 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.981178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37315 (local address 127.18.44.195:44797)
0923 18:05:55.982235 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:55.982306 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:55.983880 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:55.984803 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:55.984814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:55.984945 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:55.985703 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:55.985705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:55.985761 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:55.985863 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:55.986265 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:55.986274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:55.986281 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:05:55.986498 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 18:05:55.986835 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":196,"threads_started":1}
I20260923 18:05:55.990170 23686 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37315
I20260923 18:05:55.990522 23686 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:55.991093 23686 heartbeater.cc:511] Master 127.18.44.254:37315 requested a full tablet report, sending...
I20260923 18:05:55.992744 23235 ts_manager.cc:194] Registered new tserver with Master: 3b60e0c460324bd39ef0d1d28a10c810 (127.18.44.195:40265)
I20260923 18:05:55.993880 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:05:55.998186 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:55.996102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.18.44.254:37315)
0923 18:05:55.996329 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:55.996334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:55.996520 (+ 186us) server_negotiation.cc:407] Connection header received
0923 18:05:55.996579 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:55.996581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:55.996622 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:55.996703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:55.997609 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:55.997613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:55.997616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:55.997632 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:55.997662 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:55.997672 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:55.997776 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:55.997873 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 18:05:55.997986 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 18:05:56.002327 23235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:05:56.006443 23235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:05:56.022441 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.018601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:33505 (local address 127.0.0.1:41934)
0923 18:05:56.019236 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.019333 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.019483 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.020763 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.020770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.020859 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:56.021511 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:56.021513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:56.021532 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:56.021607 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:56.022104 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:56.022107 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:56.022111 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:05:56.022172 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 18:05:56.022302 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":155,"threads_started":1}
I20260923 18:05:56.022433 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.018624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.18.44.193:34535)
0923 18:05:56.019159 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.019165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.019840 (+ 675us) server_negotiation.cc:407] Connection header received
0923 18:05:56.020023 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.020028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.020184 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.020282 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.021829 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.021836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.021851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.021865 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.021896 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.021919 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.022016 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.022174 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.022264 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":156,"threads_started":1}
I20260923 18:05:56.022639 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.018775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.18.44.194:33505)
0923 18:05:56.019352 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.019366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.019588 (+ 222us) server_negotiation.cc:407] Connection header received
0923 18:05:56.019651 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.019657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.019885 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.020024 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.021817 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.021826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.021843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.021861 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.021929 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.021986 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.022143 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.022198 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.022295 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":135,"threads_started":1}
I20260923 18:05:56.023005 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.018628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34535 (local address 127.0.0.1:40144)
0923 18:05:56.019743 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.019757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.019895 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.020839 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.020842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.020858 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:56.021637 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:56.021639 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:56.021647 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:56.021666 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:56.021999 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:56.022008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:56.022013 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:05:56.022767 (+ 754us) client_negotiation.cc:240] Negotiation successful
0923 18:05:56.022804 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":154,"threads_started":1}
I20260923 18:05:56.024576 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.019721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40265 (local address 127.0.0.1:57652)
0923 18:05:56.020090 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.020106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.021047 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.023201 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.023204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.023223 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:56.023835 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:56.023837 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:56.023852 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:56.023884 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:56.024317 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:56.024320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:56.024324 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:05:56.024396 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:05:56.024441 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
I20260923 18:05:56.025180 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.020400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.18.44.195:40265)
0923 18:05:56.020902 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.020915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.021032 (+ 117us) server_negotiation.cc:407] Connection header received
0923 18:05:56.021701 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.021710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.022050 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.022217 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.024046 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.024053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.024067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.024080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.024132 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.024168 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.024274 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.024887 (+ 613us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.025032 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":121,"threads_started":1}
I20260923 18:05:56.026472 23358 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d5fec1c504aae80b45f75127769cd (DEFAULT_TABLE table=test-table [id=9acee066dadd40f0abb72845897e9e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:56.026472 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d5fec1c504aae80b45f75127769cd (DEFAULT_TABLE table=test-table [id=9acee066dadd40f0abb72845897e9e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:56.027727 23358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d5fec1c504aae80b45f75127769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:56.027727 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d5fec1c504aae80b45f75127769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:56.030032 23620 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d5fec1c504aae80b45f75127769cd (DEFAULT_TABLE table=test-table [id=9acee066dadd40f0abb72845897e9e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:05:56.032196 23620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d5fec1c504aae80b45f75127769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:56.037669 23703 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: Bootstrap starting.
I20260923 18:05:56.038434 23704 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Bootstrap starting.
I20260923 18:05:56.042032 23703 tablet_bootstrap.cc:654] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:56.042289 23704 tablet_bootstrap.cc:654] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:56.043165 23703 log.cc:824] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:56.043674 23705 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Bootstrap starting.
I20260923 18:05:56.043824 23704 log.cc:824] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:56.045925 23703 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: No bootstrap required, opened a new log
I20260923 18:05:56.046171 23703 ts_tablet_manager.cc:1397] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260923 18:05:56.046429 23705 tablet_bootstrap.cc:654] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:56.046420 23704 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: No bootstrap required, opened a new log
I20260923 18:05:56.046885 23704 ts_tablet_manager.cc:1397] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260923 18:05:56.047938 23705 log.cc:824] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:56.050932 23705 tablet_bootstrap.cc:492] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: No bootstrap required, opened a new log
I20260923 18:05:56.051213 23705 ts_tablet_manager.cc:1397] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 18:05:56.051355 23703 raft_consensus.cc:348] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.051715 23703 raft_consensus.cc:374] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:56.051838 23703 raft_consensus.cc:729] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74b3d6143174fd8b8e9fb2a329eab98, State: Initialized, Role: FOLLOWER
I20260923 18:05:56.051646 23704 raft_consensus.cc:348] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.052209 23704 raft_consensus.cc:374] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:56.052358 23704 raft_consensus.cc:729] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a758afbd1edc401d89fb1dcc37543a1b, State: Initialized, Role: FOLLOWER
I20260923 18:05:56.052610 23703 consensus_queue.cc:260] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.052969 23704 consensus_queue.cc:260] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.053759 23424 heartbeater.cc:503] Master 127.18.44.254:37315 was elected leader, sending a full tablet report...
I20260923 18:05:56.054070 23703 ts_tablet_manager.cc:1428] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260923 18:05:56.054455 23704 ts_tablet_manager.cc:1428] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260923 18:05:56.055001 23555 heartbeater.cc:503] Master 127.18.44.254:37315 was elected leader, sending a full tablet report...
I20260923 18:05:56.056370 23705 raft_consensus.cc:348] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.056838 23705 raft_consensus.cc:374] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:56.056900 23705 raft_consensus.cc:729] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b60e0c460324bd39ef0d1d28a10c810, State: Initialized, Role: FOLLOWER
I20260923 18:05:56.057437 23705 consensus_queue.cc:260] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.058248 23686 heartbeater.cc:503] Master 127.18.44.254:37315 was elected leader, sending a full tablet report...
I20260923 18:05:56.058678 23705 ts_tablet_manager.cc:1428] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260923 18:05:56.116616 23710 raft_consensus.cc:482] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:05:56.116864 23710 raft_consensus.cc:504] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.118196 23710 leader_election.cc:288] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74b3d6143174fd8b8e9fb2a329eab98 (127.18.44.193:34535), 3b60e0c460324bd39ef0d1d28a10c810 (127.18.44.195:40265)
I20260923 18:05:56.120466 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.118591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40265 (local address 127.18.44.194:35313)
0923 18:05:56.118858 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.118875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.118993 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.119337 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.119339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.119357 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:56.119915 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:56.119916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:56.119928 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:56.119958 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:56.120196 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:56.120199 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:56.120202 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:05:56.120278 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:05:56.120323 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260923 18:05:56.120466 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.118685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:35313 (local address 127.18.44.195:40265)
0923 18:05:56.118904 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.118910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.118939 (+ 29us) server_negotiation.cc:407] Connection header received
0923 18:05:56.119137 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.119141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.119186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.119315 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.120080 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.120084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.120086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.120096 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.120116 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.120123 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.120204 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.120279 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.120323 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 18:05:56.121223 23638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d5fec1c504aae80b45f75127769cd" candidate_uuid: "a758afbd1edc401d89fb1dcc37543a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b60e0c460324bd39ef0d1d28a10c810" is_pre_election: true
I20260923 18:05:56.121297 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.118731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34535 (local address 127.18.44.194:59917)
0923 18:05:56.119416 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.119431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.119570 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.119839 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.119843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.119859 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:56.120440 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:05:56.120442 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:56.120454 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:05:56.120485 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:56.121009 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:56.121013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:56.121016 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:05:56.121097 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:05:56.121139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":230,"thread_start_us":175,"threads_started":1}
I20260923 18:05:56.121340 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.118894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:59917 (local address 127.18.44.193:34535)
0923 18:05:56.119084 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.119089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.119559 (+ 470us) server_negotiation.cc:407] Connection header received
0923 18:05:56.119667 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.119670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.119707 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.119792 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.120773 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.120778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.120781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.120798 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.120838 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.120855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.120939 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.121142 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.121201 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 18:05:56.121575 23638 raft_consensus.cc:2454] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a758afbd1edc401d89fb1dcc37543a1b in term 0.
I20260923 18:05:56.122229 23445 leader_election.cc:302] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b60e0c460324bd39ef0d1d28a10c810, a758afbd1edc401d89fb1dcc37543a1b; no voters:
I20260923 18:05:56.122377 23378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d5fec1c504aae80b45f75127769cd" candidate_uuid: "a758afbd1edc401d89fb1dcc37543a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" is_pre_election: true
I20260923 18:05:56.122706 23710 raft_consensus.cc:2781] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:05:56.122788 23378 raft_consensus.cc:2454] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a758afbd1edc401d89fb1dcc37543a1b in term 0.
I20260923 18:05:56.122833 23710 raft_consensus.cc:482] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:05:56.122895 23710 raft_consensus.cc:3037] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:56.124188 23710 raft_consensus.cc:504] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.124625 23710 leader_election.cc:288] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [CANDIDATE]: Term 1 election: Requested vote from peers a74b3d6143174fd8b8e9fb2a329eab98 (127.18.44.193:34535), 3b60e0c460324bd39ef0d1d28a10c810 (127.18.44.195:40265)
I20260923 18:05:56.125384 23638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d5fec1c504aae80b45f75127769cd" candidate_uuid: "a758afbd1edc401d89fb1dcc37543a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b60e0c460324bd39ef0d1d28a10c810"
I20260923 18:05:56.125447 23378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d5fec1c504aae80b45f75127769cd" candidate_uuid: "a758afbd1edc401d89fb1dcc37543a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74b3d6143174fd8b8e9fb2a329eab98"
I20260923 18:05:56.125566 23638 raft_consensus.cc:3037] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:56.125595 23378 raft_consensus.cc:3037] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:56.126955 23378 raft_consensus.cc:2454] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a758afbd1edc401d89fb1dcc37543a1b in term 1.
I20260923 18:05:56.126950 23638 raft_consensus.cc:2454] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a758afbd1edc401d89fb1dcc37543a1b in term 1.
I20260923 18:05:56.127458 23445 leader_election.cc:302] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b60e0c460324bd39ef0d1d28a10c810, a758afbd1edc401d89fb1dcc37543a1b; no voters:
I20260923 18:05:56.127751 23710 raft_consensus.cc:2781] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:56.128012 23710 raft_consensus.cc:686] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 1 LEADER]: Becoming Leader. State: Replica: a758afbd1edc401d89fb1dcc37543a1b, State: Running, Role: LEADER
I20260923 18:05:56.128427 23710 consensus_queue.cc:237] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } }
I20260923 18:05:56.132025 23234 catalog_manager.cc:5495] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a758afbd1edc401d89fb1dcc37543a1b (127.18.44.194). New cstate: current_term: 1 leader_uuid: "a758afbd1edc401d89fb1dcc37543a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a758afbd1edc401d89fb1dcc37543a1b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 33505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 } health_report { overall_health: UNKNOWN } } }
I20260923 18:05:56.159576 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.157667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.18.44.194:33505)
0923 18:05:56.157822 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.157826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.157843 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:05:56.158082 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.158086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.158178 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.158277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.159115 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.159118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.159121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.159134 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.159155 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.159164 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.159296 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.159372 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.159423 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:05:56.167570 23638 raft_consensus.cc:1261] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 1 FOLLOWER]: Refusing update from remote peer a758afbd1edc401d89fb1dcc37543a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:56.167623 23378 raft_consensus.cc:1261] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98 [term 1 FOLLOWER]: Refusing update from remote peer a758afbd1edc401d89fb1dcc37543a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:05:56.168568 23713 consensus_queue.cc:1035] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b60e0c460324bd39ef0d1d28a10c810" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:56.168838 23710 consensus_queue.cc:1035] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74b3d6143174fd8b8e9fb2a329eab98" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:05:56.192168 23235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:56.192476 23235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:05:56.196353 23235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d51e4cc402fa4478a8dfdf993b738cbd: Sending DeleteTablet for 3 replicas of tablet 0b4d5fec1c504aae80b45f75127769cd
I20260923 18:05:56.197559 23358 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d5fec1c504aae80b45f75127769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:56 UTC) from {username='slave'} at 127.0.0.1:40144
I20260923 18:05:56.197705 23489 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d5fec1c504aae80b45f75127769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:56 UTC) from {username='slave'} at 127.0.0.1:41934
I20260923 18:05:56.198566 23727 tablet_replica.cc:330] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: stopping tablet replica
I20260923 18:05:56.198715 23620 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d5fec1c504aae80b45f75127769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:05:56 UTC) from {username='slave'} at 127.0.0.1:57652
I20260923 18:05:56.199169 23727 raft_consensus.cc:2229] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:05:56.199605 23727 raft_consensus.cc:2258] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:05:56.200028 23726 tablet_replica.cc:330] T 0b4d5fec1c504aae80b45f75127769cd P a74b3d6143174fd8b8e9fb2a329eab98: stopping tablet replica
I20260923 18:05:56.200285 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23296
I20260923 18:05:56.203130 23727 ts_tablet_manager.cc:1905] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:56.205713 23727 ts_tablet_manager.cc:1918] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:56.205832 23727 log.cc:1196] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/0b4d5fec1c504aae80b45f75127769cd
I20260923 18:05:56.206225 23727 ts_tablet_manager.cc:1939] T 0b4d5fec1c504aae80b45f75127769cd P a758afbd1edc401d89fb1dcc37543a1b: Deleting consensus metadata
I20260923 18:05:56.207911 23223 catalog_manager.cc:4841] TS a758afbd1edc401d89fb1dcc37543a1b (127.18.44.194:33505): tablet 0b4d5fec1c504aae80b45f75127769cd (table test-table [id=9acee066dadd40f0abb72845897e9e3b]) successfully deleted
I20260923 18:05:56.209416 23728 tablet_replica.cc:330] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: stopping tablet replica
I20260923 18:05:56.209766 23728 raft_consensus.cc:2229] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:05:56.210197 23728 raft_consensus.cc:2258] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:05:56.210621 23220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:34535 (error 108)
I20260923 18:05:56.211855 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23427
I20260923 18:05:56.213529 23728 ts_tablet_manager.cc:1905] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:05:56.216332 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.215891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34535 (local address 127.0.0.1:40146)
0923 18:05:56.216081 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.216193 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 18:05:56.216516 23728 ts_tablet_manager.cc:1918] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:05:56.216645 23728 log.cc:1196] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/0b4d5fec1c504aae80b45f75127769cd
W20260923 18:05:56.216720 23220 catalog_manager.cc:4584] TS a74b3d6143174fd8b8e9fb2a329eab98 (127.18.44.193:34535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b4d5fec1c504aae80b45f75127769cd: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34535: connect: Connection refused (error 111)
I20260923 18:05:56.217021 23728 ts_tablet_manager.cc:1939] T 0b4d5fec1c504aae80b45f75127769cd P 3b60e0c460324bd39ef0d1d28a10c810: Deleting consensus metadata
I20260923 18:05:56.218470 23223 catalog_manager.cc:4841] TS 3b60e0c460324bd39ef0d1d28a10c810 (127.18.44.195:40265): tablet 0b4d5fec1c504aae80b45f75127769cd (table test-table [id=9acee066dadd40f0abb72845897e9e3b]) successfully deleted
I20260923 18:05:56.222057 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23558
I20260923 18:05:56.228055 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23205
2026-09-23T18:05:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1284 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 18:05:56.252225 18611 mini_oidc.cc:152] Starting JWKS server
I20260923 18:05:56.254251 18611 webserver.cc:460] Webserver started at https://127.0.0.1:42557/ using document root <none> and password file <none>
I20260923 18:05:56.254467 18611 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:05:56.254781 18611 webserver.cc:460] Webserver started at http://127.0.0.1:38573/ using document root <none> and password file <none>
2026-09-23T18:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:56Z Disabled control of system clock
I20260923 18:05:56.266201 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45645
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:46695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/testchainca.cert with env {}
W20260923 18:05:56.407289 23735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:56.407729 23735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:56.407779 23735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:56.413849 23735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:56.413980 23735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:56.413997 23735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:56.414016 23735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:56.414032 23735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:05:56.414049 23735 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:05:56.414079 23735 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42557/jwks.json
I20260923 18:05:56.421221 23735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:56.422799 23735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:56.424830 23735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:56.437999 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:56.438010 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:56.437999 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:56.438344 23735 server_base.cc:1034] running on GCE node
I20260923 18:05:56.439447 23735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:56.440790 23735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:56.442010 23735 hybrid_clock.cc:648] HybridClock initialized: now 1790186756441985 us; error 55 us; skew 500 ppm
I20260923 18:05:56.444587 23735 webserver.cc:460] Webserver started at http://127.18.44.254:39653/ using document root <none> and password file <none>
I20260923 18:05:56.445358 23735 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:56.445550 23735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:56.445981 23735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:56.448485 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "289a2d2910794f8088bda21d200de7e5"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "da20bb58241068661d287554a7e9ccb4"
server_key_iv: "6a56e8ec1b0af04cf7c2cbf6be4b904a"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.449090 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "289a2d2910794f8088bda21d200de7e5"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "da20bb58241068661d287554a7e9ccb4"
server_key_iv: "6a56e8ec1b0af04cf7c2cbf6be4b904a"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.453737 23735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:05:56.457167 23749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:56.459000 23735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:05:56.459167 23735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "289a2d2910794f8088bda21d200de7e5"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "da20bb58241068661d287554a7e9ccb4"
server_key_iv: "6a56e8ec1b0af04cf7c2cbf6be4b904a"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.459383 23735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:56.509425 23735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:56.510264 23735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:56.510456 23735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:56.518895 23735 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45645
I20260923 18:05:56.518906 23801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45645 every 8 connection(s)
I20260923 18:05:56.520054 23735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:56.524904 23802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:56.525695 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23735
I20260923 18:05:56.525923 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:56.526306 18611 external_mini_cluster.cc:1428] Setting key f00a91720e3a424c37025f7e8dc3e69e
I20260923 18:05:56.534248 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: Bootstrap starting.
I20260923 18:05:56.538043 23802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:56.539194 23802 log.cc:824] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:56.539207 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.528373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.18.44.254:45645)
0923 18:05:56.529149 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.529168 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.529211 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:05:56.530071 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.530095 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.530597 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.530965 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.531973 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.533202 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.534710 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.535363 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.535544 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.536815 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.536821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.536839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.537418 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.537520 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.537546 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.537647 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.537896 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.538253 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":145,"threads_started":1}
I20260923 18:05:56.542407 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: No bootstrap required, opened a new log
I20260923 18:05:56.543598 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:05:56.547379 23802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } }
I20260923 18:05:56.547976 23802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:56.548060 23802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289a2d2910794f8088bda21d200de7e5, State: Initialized, Role: FOLLOWER
I20260923 18:05:56.548926 23802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } }
I20260923 18:05:56.549180 23802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:56.549266 23802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:56.549408 23802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:56.550817 23802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } }
I20260923 18:05:56.551431 23802 leader_election.cc:302] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289a2d2910794f8088bda21d200de7e5; no voters:
I20260923 18:05:56.551915 23802 leader_election.cc:288] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:56.552146 23807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:56.552627 23807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 289a2d2910794f8088bda21d200de7e5, State: Running, Role: LEADER
I20260923 18:05:56.553149 23807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } }
I20260923 18:05:56.553812 23802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:56.555383 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289a2d2910794f8088bda21d200de7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } } }
I20260923 18:05:56.555646 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:56.556070 23809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289a2d2910794f8088bda21d200de7e5. Latest consensus state: current_term: 1 leader_uuid: "289a2d2910794f8088bda21d200de7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289a2d2910794f8088bda21d200de7e5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45645 } } }
I20260923 18:05:56.556329 23809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5 [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:56.560286 23822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:56.560415 23822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:56.560498 23816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:56.561244 23816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:56.568617 23816 catalog_manager.cc:1347] Generated new cluster ID: f77f10db86f24555bbdc50bb874f1e4b
I20260923 18:05:56.568727 23816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:56.586894 23816 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:56.588771 23816 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:56.598357 23816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 289a2d2910794f8088bda21d200de7e5: Generated new TSK 0
I20260923 18:05:56.599555 23816 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 18:05:56.645293 23832 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 18:05:56.645411 23832 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 18:05:56.646579 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.624774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.18.44.254:45645)
0923 18:05:56.625019 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.625025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.625146 (+ 121us) server_negotiation.cc:407] Connection header received
0923 18:05:56.625534 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.625538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.625626 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.625796 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:05:56.626183 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.627101 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.628403 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.628830 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.628997 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.629518 (+ 521us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:05:56.645877 (+ 16359us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 18:05:56.646351 (+ 474us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":106}
W20260923 18:05:56.646703 23805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 18:05:56.648134 23831 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:05:56.624559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45645 (local address 127.0.0.1:51804)
0923 18:05:56.625037 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.625055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.625292 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.625902 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.625905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.625914 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:05:56.626058 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:56.626064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.627406 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.627409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:56.628197 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:56.628210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.629030 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.629033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:56.629134 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.629211 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:05:56.646385 (+ 17174us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 18:05:56.646803 (+ 418us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.44.254:45645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.44.254:45645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 18:05:56.649662 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23735
2026-09-23T18:05:56Z chronyd exiting
I20260923 18:05:56.688445 18611 test_util.cc:175] -----------------------------------------------
I20260923 18:05:56.688637 18611 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790186727112038-18611-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (438 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 18:05:56.690135 18611 mini_oidc.cc:152] Starting JWKS server
I20260923 18:05:56.691161 18611 webserver.cc:460] Webserver started at https://127.0.0.1:44955/ using document root <none> and password file <none>
I20260923 18:05:56.691293 18611 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:05:56.691565 18611 webserver.cc:460] Webserver started at http://127.0.0.1:45611/ using document root <none> and password file <none>
2026-09-23T18:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:56Z Disabled control of system clock
I20260923 18:05:56.702384 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37193
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44955/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/testchainca.cert with env {}
W20260923 18:05:56.836520 23838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:56.836865 23838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:56.836925 23838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:56.841277 23838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:56.841370 23838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:56.841391 23838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:56.841413 23838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:56.841431 23838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:05:56.841454 23838 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:05:56.841472 23838 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44955/jwks.json
I20260923 18:05:56.847479 23838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44955/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:56.848695 23838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:56.849794 23838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:56.859691 23846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:56.859824 23844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:56.859691 23843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:56.859694 23838 server_base.cc:1034] running on GCE node
I20260923 18:05:56.861618 23838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:56.862941 23838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:56.866322 23838 hybrid_clock.cc:648] HybridClock initialized: now 1790186756866260 us; error 129 us; skew 500 ppm
I20260923 18:05:56.868803 23838 webserver.cc:460] Webserver started at http://127.18.44.254:39221/ using document root <none> and password file <none>
I20260923 18:05:56.869446 23838 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:56.869527 23838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:56.869822 23838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:56.872159 23838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "310f836a2cc3469084cead51fe91a2d0"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "b8f3acf053fec65e4d8e0809bd048e03"
server_key_iv: "bcb54dc4195af91ac4f31335737b519d"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.872821 23838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "310f836a2cc3469084cead51fe91a2d0"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "b8f3acf053fec65e4d8e0809bd048e03"
server_key_iv: "bcb54dc4195af91ac4f31335737b519d"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.878013 23838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:05:56.880924 23852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:56.882319 23838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:05:56.882470 23838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "310f836a2cc3469084cead51fe91a2d0"
format_stamp: "Formatted at 2026-09-23 18:05:56 on dist-test-slave-rvjj"
server_key: "b8f3acf053fec65e4d8e0809bd048e03"
server_key_iv: "bcb54dc4195af91ac4f31335737b519d"
server_key_version: "encryptionkey@0"
I20260923 18:05:56.882688 23838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:56.894807 23838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:56.895606 23838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:56.895800 23838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:56.903019 23838 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37193
I20260923 18:05:56.903044 23904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37193 every 8 connection(s)
I20260923 18:05:56.904354 23838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:56.908109 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:56.912211 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23838
I20260923 18:05:56.912364 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:56.912640 18611 external_mini_cluster.cc:1428] Setting key 92d986da79d4ec7467a42223972ea429
I20260923 18:05:56.914846 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0: Bootstrap starting.
I20260923 18:05:56.917373 23905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:56.918126 23905 log.cc:824] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:56.920190 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0: No bootstrap required, opened a new log
I20260923 18:05:56.922324 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.914205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.18.44.254:37193)
0923 18:05:56.914920 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.914940 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.914988 (+ 48us) server_negotiation.cc:407] Connection header received
0923 18:05:56.915779 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.915799 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.916265 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.916596 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:56.917046 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.918105 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.919048 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.919583 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.919746 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.920589 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:56.920595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:56.920608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:56.921027 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:56.921083 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:56.921104 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:56.921197 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:56.921318 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 18:05:56.921588 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":150,"threads_started":1}
I20260923 18:05:56.924182 23905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } }
I20260923 18:05:56.924477 23905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:56.924547 23905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310f836a2cc3469084cead51fe91a2d0, State: Initialized, Role: FOLLOWER
I20260923 18:05:56.924991 23905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } }
I20260923 18:05:56.925143 23905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:56.925208 23905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:56.925302 23905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:56.925563 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:05:56.926565 23905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } }
I20260923 18:05:56.926944 23905 leader_election.cc:302] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310f836a2cc3469084cead51fe91a2d0; no voters:
I20260923 18:05:56.927332 23905 leader_election.cc:288] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:56.927551 23910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:56.927780 23910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 310f836a2cc3469084cead51fe91a2d0, State: Running, Role: LEADER
I20260923 18:05:56.928110 23910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } }
I20260923 18:05:56.928922 23905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:56.929653 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310f836a2cc3469084cead51fe91a2d0. Latest consensus state: current_term: 1 leader_uuid: "310f836a2cc3469084cead51fe91a2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } } }
I20260923 18:05:56.930003 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310f836a2cc3469084cead51fe91a2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310f836a2cc3469084cead51fe91a2d0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37193 } } }
I20260923 18:05:56.930094 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:56.929802 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:56.930518 23919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:56.933933 23919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:56.941322 23919 catalog_manager.cc:1347] Generated new cluster ID: 7eee5d7f73fb4530b144c7add0b39e28
I20260923 18:05:56.941418 23919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:56.957876 23919 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:56.958866 23919 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:56.963984 23919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310f836a2cc3469084cead51fe91a2d0: Generated new TSK 0
I20260923 18:05:56.964907 23919 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 18:05:56.999475 23935 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 18:05:56.999562 23935 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 18:05:57.000353 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:56.981090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.18.44.254:37193)
0923 18:05:56.981252 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:56.981257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:56.981355 (+ 98us) server_negotiation.cc:407] Connection header received
0923 18:05:56.981526 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:56.981529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:56.981592 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:56.981728 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:05:56.982118 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.983548 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.984810 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.985174 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.985290 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.985752 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:05:56.999880 (+ 14128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 18:05:57.000188 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 18:05:57.000428 23908 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 18:05:57.000591 23934 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:05:56.980913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37193 (local address 127.0.0.1:58182)
0923 18:05:56.981262 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:56.981278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:56.981390 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:56.981817 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:56.981820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:56.981828 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:05:56.981949 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:56.981955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.983898 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.983903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:56.984617 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:56.984628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:56.985339 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:56.985342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:56.985430 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:56.985474 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:05:57.000108 (+ 14634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 18:05:57.000289 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.44.254:37193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
I20260923 18:05:57.001317 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23838
2026-09-23T18:05:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 18:05:57.040896 18611 mini_oidc.cc:152] Starting JWKS server
I20260923 18:05:57.041805 18611 webserver.cc:460] Webserver started at https://127.0.0.1:41785/ using document root <none> and password file <none>
I20260923 18:05:57.041903 18611 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:05:57.042160 18611 webserver.cc:460] Webserver started at http://127.0.0.1:41705/ using document root <none> and password file <none>
2026-09-23T18:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:57Z Disabled control of system clock
I20260923 18:05:57.051524 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41997
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:41997
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 18:05:57.173022 23941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:57.173343 23941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:57.173385 23941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:57.177821 23941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:57.177901 23941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:57.177915 23941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:57.177932 23941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:57.177958 23941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:05:57.177977 23941 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:05:57.178006 23941 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41785/jwks.json
I20260923 18:05:57.183120 23941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:41997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:57.184316 23941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:57.185801 23941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:57.195235 23949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:57.195307 23941 server_base.cc:1034] running on GCE node
W20260923 18:05:57.195222 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:57.195222 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:57.196197 23941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:57.197399 23941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:57.198598 23941 hybrid_clock.cc:648] HybridClock initialized: now 1790186757198577 us; error 47 us; skew 500 ppm
I20260923 18:05:57.200831 23941 webserver.cc:460] Webserver started at http://127.18.44.254:36427/ using document root <none> and password file <none>
I20260923 18:05:57.201432 23941 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:57.201501 23941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:57.201684 23941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:57.203949 23941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "e42f07998e404f748e74a1a8c738eff1"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "ff243ce1a73e848448c14fe8049fd97a"
server_key_iv: "e79d208af7c578d17061ad325dc6cac8"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.204524 23941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "e42f07998e404f748e74a1a8c738eff1"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "ff243ce1a73e848448c14fe8049fd97a"
server_key_iv: "e79d208af7c578d17061ad325dc6cac8"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.208645 23941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 18:05:57.211490 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:57.212976 23941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:57.213136 23941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "e42f07998e404f748e74a1a8c738eff1"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "ff243ce1a73e848448c14fe8049fd97a"
server_key_iv: "e79d208af7c578d17061ad325dc6cac8"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.213263 23941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:57.239058 23941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:57.239863 23941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:57.240053 23941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:57.247200 23941 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:41997
I20260923 18:05:57.247220 24007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:41997 every 8 connection(s)
I20260923 18:05:57.248286 23941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:57.251538 24008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:57.257400 24008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: Bootstrap starting.
I20260923 18:05:57.258616 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 23941
I20260923 18:05:57.258733 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:57.258991 18611 external_mini_cluster.cc:1428] Setting key d50e16cb8d14aeae62eb65c22eb5f350
I20260923 18:05:57.260248 24008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:57.261164 24008 log.cc:824] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:57.263654 24008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: No bootstrap required, opened a new log
I20260923 18:05:57.267539 24008 raft_consensus.cc:348] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } }
I20260923 18:05:57.267910 24008 raft_consensus.cc:374] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:57.268007 24008 raft_consensus.cc:729] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42f07998e404f748e74a1a8c738eff1, State: Initialized, Role: FOLLOWER
I20260923 18:05:57.268571 24008 consensus_queue.cc:260] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } }
I20260923 18:05:57.268730 24008 raft_consensus.cc:388] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:57.268796 24008 raft_consensus.cc:482] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:57.268899 24008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:57.268934 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:57.260236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.18.44.254:41997)
0923 18:05:57.261028 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:57.261044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:57.261082 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:05:57.261930 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:57.261957 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:57.262398 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:57.262742 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:57.263198 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.264320 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.265276 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.265797 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.265961 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.266787 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:57.266792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:57.266806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:57.267238 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:57.267355 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:57.267379 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:57.267467 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:57.267612 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 18:05:57.267927 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":175,"threads_started":1}
I20260923 18:05:57.270211 24008 raft_consensus.cc:504] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } }
I20260923 18:05:57.270591 24008 leader_election.cc:302] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e42f07998e404f748e74a1a8c738eff1; no voters:
I20260923 18:05:57.270939 24008 leader_election.cc:288] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:57.271039 24013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:57.271215 24013 raft_consensus.cc:686] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [term 1 LEADER]: Becoming Leader. State: Replica: e42f07998e404f748e74a1a8c738eff1, State: Running, Role: LEADER
I20260923 18:05:57.271572 24013 consensus_queue.cc:237] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } }
I20260923 18:05:57.272437 24008 sys_catalog.cc:564] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:57.272754 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:05:57.273507 24014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e42f07998e404f748e74a1a8c738eff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } } }
I20260923 18:05:57.273671 24014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:57.274005 24015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e42f07998e404f748e74a1a8c738eff1. Latest consensus state: current_term: 1 leader_uuid: "e42f07998e404f748e74a1a8c738eff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42f07998e404f748e74a1a8c738eff1" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41997 } } }
I20260923 18:05:57.274112 24015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1 [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:57.277740 24029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:57.277853 24029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:57.277961 24020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:57.278784 24020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:57.285769 24020 catalog_manager.cc:1347] Generated new cluster ID: fa0e2e4c67b4489488ed28842703ea5c
I20260923 18:05:57.285866 24020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:57.297807 24020 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:57.300643 24020 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:57.310093 24020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e42f07998e404f748e74a1a8c738eff1: Generated new TSK 0
I20260923 18:05:57.311101 24020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:05:57.336298 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:57.329713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.18.44.254:41997)
0923 18:05:57.329854 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:57.329859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:57.329939 (+ 80us) server_negotiation.cc:407] Connection header received
0923 18:05:57.330166 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:57.330178 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:57.330323 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:57.330623 (+ 300us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:05:57.330874 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.331943 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.333289 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.333660 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.333801 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.334164 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:05:57.335841 (+ 1677us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 18:05:57.336121 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 18:05:57.336385 24011 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 18:05:57.337981 24037 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:05:57.329547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41997 (local address 127.0.0.1:32820)
0923 18:05:57.329847 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:57.329861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:57.329970 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:57.330496 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:57.330499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:57.330506 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:05:57.330696 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:57.330707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.332128 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.332133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:57.333036 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:57.333049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.333834 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.333837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:57.333916 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.333945 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:05:57.336055 (+ 2110us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 18:05:57.337777 (+ 1722us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.44.254:41997: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":1467,"thread_start_us":107,"threads_started":1}
I20260923 18:05:57.338645 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 23941
E20260923 18:05:57.349936 24038 webserver.cc:533] Webserver: sslize failed:
E20260923 18:05:57.350047 24038 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T18:05:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T18:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:57Z Disabled control of system clock
I20260923 18:05:57.389389 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45739
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45739
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/testchainca.cert with env {}
W20260923 18:05:57.512132 24042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:57.512498 24042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:57.512550 24042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:57.517663 24042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:57.517782 24042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:57.517798 24042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:57.517817 24042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:57.517831 24042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:05:57.517848 24042 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:05:57.517860 24042 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 18:05:57.524989 24042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:57.526247 24042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:57.527742 24042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:57.537446 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:57.537576 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:57.537568 24042 server_base.cc:1034] running on GCE node
W20260923 18:05:57.537446 24050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:57.538535 24042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:57.539921 24042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:57.541184 24042 hybrid_clock.cc:648] HybridClock initialized: now 1790186757541165 us; error 39 us; skew 500 ppm
I20260923 18:05:57.543606 24042 webserver.cc:460] Webserver started at http://127.18.44.254:35751/ using document root <none> and password file <none>
I20260923 18:05:57.544220 24042 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:57.544308 24042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:57.544535 24042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:57.547009 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "6777eee5549c438e895150d97b319932"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "50e798ac00dcd156f11bec590b8d3948"
server_key_iv: "1d5ae649545c40d23385ab5f33ba70dd"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.547786 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "6777eee5549c438e895150d97b319932"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "50e798ac00dcd156f11bec590b8d3948"
server_key_iv: "1d5ae649545c40d23385ab5f33ba70dd"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.552304 24042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:05:57.555543 24056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:57.557120 24042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:05:57.557288 24042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "6777eee5549c438e895150d97b319932"
format_stamp: "Formatted at 2026-09-23 18:05:57 on dist-test-slave-rvjj"
server_key: "50e798ac00dcd156f11bec590b8d3948"
server_key_iv: "1d5ae649545c40d23385ab5f33ba70dd"
server_key_version: "encryptionkey@0"
I20260923 18:05:57.557428 24042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:57.591881 24042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:57.592676 24042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:57.593015 24042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:57.601322 24042 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45739
I20260923 18:05:57.601354 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45739 every 8 connection(s)
I20260923 18:05:57.602460 24042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:57.605669 24109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:57.607431 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24042
I20260923 18:05:57.607586 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:57.607892 18611 external_mini_cluster.cc:1428] Setting key 7acdb2862af6fb7cdb31c67321a71362
I20260923 18:05:57.613211 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932: Bootstrap starting.
I20260923 18:05:57.615973 24109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:57.616802 24109 log.cc:824] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:57.618832 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:57.609514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.18.44.254:45739)
0923 18:05:57.610290 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:57.610314 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:57.610358 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:05:57.611123 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:57.611149 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:57.611720 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:57.612135 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:57.612594 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.614025 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.615098 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.615854 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.616030 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.616907 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:57.616914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:57.616932 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:05:57.617438 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:57.617508 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:05:57.617532 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:57.617631 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:57.617792 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 18:05:57.618070 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260923 18:05:57.619328 24109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932: No bootstrap required, opened a new log
I20260923 18:05:57.622107 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:05:57.623934 24109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } }
I20260923 18:05:57.624292 24109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:57.624413 24109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6777eee5549c438e895150d97b319932, State: Initialized, Role: FOLLOWER
I20260923 18:05:57.625062 24109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } }
I20260923 18:05:57.625428 24109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:57.625514 24109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:57.625631 24109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:57.626981 24109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } }
I20260923 18:05:57.627481 24109 leader_election.cc:302] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6777eee5549c438e895150d97b319932; no voters:
I20260923 18:05:57.628108 24109 leader_election.cc:288] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:57.628201 24114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:57.628455 24114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [term 1 LEADER]: Becoming Leader. State: Replica: 6777eee5549c438e895150d97b319932, State: Running, Role: LEADER
I20260923 18:05:57.628795 24114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } }
I20260923 18:05:57.629487 24109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:57.630712 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6777eee5549c438e895150d97b319932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } } }
I20260923 18:05:57.630897 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:57.631153 24116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6777eee5549c438e895150d97b319932. Latest consensus state: current_term: 1 leader_uuid: "6777eee5549c438e895150d97b319932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6777eee5549c438e895150d97b319932" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45739 } } }
I20260923 18:05:57.631310 24116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:57.631420 24123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:57.634441 24123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:57.641130 24123 catalog_manager.cc:1347] Generated new cluster ID: 57bec2ffe8cd4a94820e50ef335a1452
I20260923 18:05:57.641249 24123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:57.662163 24123 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:57.663218 24123 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:57.673229 24123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6777eee5549c438e895150d97b319932: Generated new TSK 0
I20260923 18:05:57.674024 24123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:05:57.821131 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:57.685630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.18.44.254:45739)
0923 18:05:57.685770 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:57.685776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:57.685906 (+ 130us) server_negotiation.cc:407] Connection header received
0923 18:05:57.686085 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:57.686087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:57.686156 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:57.686274 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:05:57.686582 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.687191 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.688574 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.688826 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.688920 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.689441 (+ 521us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:05:57.820639 (+131198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 18:05:57.820955 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 18:05:57.821225 24112 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 18:05:57.821305 24138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:05:57.685462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45739 (local address 127.0.0.1:44716)
0923 18:05:57.685780 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:57.685796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:57.685929 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:57.686293 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:57.686297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:57.686304 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:05:57.686428 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:57.686434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.687362 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.687366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:57.688394 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:57.688408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:57.688961 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:57.688965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:57.689070 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:57.689106 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:05:57.820902 (+131796us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 18:05:57.821119 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.44.254:45739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260923 18:05:57.821889 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24042
2026-09-23T18:05:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (461 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (163 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:05:58 dist-test-slave-rvjj krb5kdc[24150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:05:58 dist-test-slave-rvjj krb5kdc[24150](info): set up 2 sockets
Sep 23 18:05:58 dist-test-slave-rvjj krb5kdc[24150](info): commencing operation
krb5kdc: starting...
W20260923 18:05:59.157024 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:05:59 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:05:59.301719 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41181
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:59.437427 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:59.437764 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:59.437836 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:59.442256 24163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:05:59.442360 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:59.442389 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:59.442409 24163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:05:59.442428 24163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:05:59.449545 24163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:59.450995 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:59.452286 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:59.463940 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:59.464102 24163 server_base.cc:1034] running on GCE node
W20260923 18:05:59.464025 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:59.464025 24168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:59.464893 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:59.466156 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:59.467352 24163 hybrid_clock.cc:648] HybridClock initialized: now 1790186759467326 us; error 37 us; skew 500 ppm
Sep 23 18:05:59 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186759, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:59.476431 24163 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:05:59.477588 24163 webserver.cc:460] Webserver started at http://127.18.44.254:34299/ using document root <none> and password file <none>
I20260923 18:05:59.478179 24163 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:59.478250 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:59.478430 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:59.480657 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "2f05df8992fd46fea3313b2f19c960c9"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "7fd0e7bcc5611e0d7d7923409cc001e3"
server_key_iv: "e316a437e9d9498fc9fb358409c3fe87"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.481249 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "2f05df8992fd46fea3313b2f19c960c9"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "7fd0e7bcc5611e0d7d7923409cc001e3"
server_key_iv: "e316a437e9d9498fc9fb358409c3fe87"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.486323 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:05:59.489368 24178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:59.491535 24163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:05:59.491708 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "2f05df8992fd46fea3313b2f19c960c9"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "7fd0e7bcc5611e0d7d7923409cc001e3"
server_key_iv: "e316a437e9d9498fc9fb358409c3fe87"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.491909 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:59.510336 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:59.515944 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:59.516155 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:59.525147 24163 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:41181
I20260923 18:05:59.525288 24230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:41181 every 8 connection(s)
I20260923 18:05:59.526499 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:05:59.530285 24231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:05:59.530282 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24163
I20260923 18:05:59.530428 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:05:59.530748 18611 external_mini_cluster.cc:1428] Setting key 55facd96ef4b34275753096ab6ea2bc9
I20260923 18:05:59.540021 24231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: Bootstrap starting.
Sep 23 18:05:59 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:05:59.542837 24231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: Neither blocks nor log segments found. Creating new log.
I20260923 18:05:59.543850 24231 log.cc:824] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:05:59.546698 24231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: No bootstrap required, opened a new log
I20260923 18:05:59.549995 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:59.532849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.18.44.254:41181)
0923 18:05:59.533451 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:59.533468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:59.533510 (+ 42us) server_negotiation.cc:407] Connection header received
0923 18:05:59.534813 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:59.534839 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:59.535243 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:59.535679 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:59.536564 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.537672 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.538597 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.539123 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.539326 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:59.542457 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:59.542464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:59.542481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:59.542518 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:59.544734 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:59.545286 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:59.545292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:59.545298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:59.545442 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:59.545728 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:59.545731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:59.545733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:59.546192 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:59.546386 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:59.546952 (+ 566us) server_negotiation.cc:299] Negotiation successful
0923 18:05:59.547567 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":179,"threads_started":1}
I20260923 18:05:59.552369 24231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } }
I20260923 18:05:59.552662 24231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:05:59.552718 24231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f05df8992fd46fea3313b2f19c960c9, State: Initialized, Role: FOLLOWER
I20260923 18:05:59.553170 24231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } }
I20260923 18:05:59.553346 24231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:05:59.553426 24231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:05:59.553519 24231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:05:59.554564 24231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } }
I20260923 18:05:59.554944 24231 leader_election.cc:302] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f05df8992fd46fea3313b2f19c960c9; no voters:
I20260923 18:05:59.555322 24231 leader_election.cc:288] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:05:59.555536 24236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:05:59.555953 24236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f05df8992fd46fea3313b2f19c960c9, State: Running, Role: LEADER
I20260923 18:05:59.556839 24236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } }
I20260923 18:05:59.557305 24231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:05:59.560137 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f05df8992fd46fea3313b2f19c960c9. Latest consensus state: current_term: 1 leader_uuid: "2f05df8992fd46fea3313b2f19c960c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } } }
I20260923 18:05:59.560361 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:05:59.560532 24237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f05df8992fd46fea3313b2f19c960c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f05df8992fd46fea3313b2f19c960c9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41181 } } }
I20260923 18:05:59.560643 24237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9 [sys.catalog]: This master's current role is: LEADER
W20260923 18:05:59.564416 24250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:05:59.564637 24250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:05:59.564733 24251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:05:59.566020 24251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:05:59.574390 24251 catalog_manager.cc:1347] Generated new cluster ID: aa9b6d3aa8b54130b02a3a88214130ee
I20260923 18:05:59.574501 24251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:05:59.599577 24251 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:05:59.601162 24251 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:05:59.621881 24251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f05df8992fd46fea3313b2f19c960c9: Generated new TSK 0
I20260923 18:05:59.623001 24251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:05:59.671101 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:05:59.789839 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:05:59.790138 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:05:59.790182 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:05:59.794473 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:05:59.794574 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:05:59.794637 24257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:05:59.800580 24257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:05:59.801961 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:05:59.803179 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:05:59.814110 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:59.814111 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:05:59.814111 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:05:59.814443 24257 server_base.cc:1034] running on GCE node
I20260923 18:05:59.815042 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:05:59.815857 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:05:59.817071 24257 hybrid_clock.cc:648] HybridClock initialized: now 1790186759817043 us; error 43 us; skew 500 ppm
Sep 23 18:05:59 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186759, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:05:59.820626 24257 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:05:59.821683 24257 webserver.cc:460] Webserver started at http://127.18.44.193:37937/ using document root <none> and password file <none>
I20260923 18:05:59.822273 24257 fs_manager.cc:359] Metadata directory not provided
I20260923 18:05:59.822350 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:05:59.822537 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:05:59.824898 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "782a5c517a004553b68442ca32c48e08"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "199a6be7f1b806d8e0c67437f5cad403"
server_key_iv: "65f2897b2648dd35a010b3ce5ea256bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.825527 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "782a5c517a004553b68442ca32c48e08"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "199a6be7f1b806d8e0c67437f5cad403"
server_key_iv: "65f2897b2648dd35a010b3ce5ea256bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.830020 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:05:59.833029 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:59.834705 24257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:05:59.834867 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "782a5c517a004553b68442ca32c48e08"
format_stamp: "Formatted at 2026-09-23 18:05:59 on dist-test-slave-rvjj"
server_key: "199a6be7f1b806d8e0c67437f5cad403"
server_key_iv: "65f2897b2648dd35a010b3ce5ea256bf"
server_key_version: "encryptionkey@0"
I20260923 18:05:59.834997 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:05:59.877882 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:05:59.881702 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:05:59.881896 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:05:59.882496 24257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:05:59.883447 24257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:05:59.883503 24257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:59.883550 24257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:05:59.883574 24257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:05:59.893997 24257 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:36255
I20260923 18:05:59.894009 24385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:36255 every 8 connection(s)
I20260923 18:05:59.895002 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:05:59.898897 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24257
I20260923 18:05:59.899024 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:05:59.899335 18611 external_mini_cluster.cc:1428] Setting key 33b041cddb922cf2caec5e1ddfe0fe29
Sep 23 18:05:59 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186759, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:05:59.910882 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:59.896902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:58505 (local address 127.18.44.254:41181)
0923 18:05:59.897069 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 18:05:59.897073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:05:59.898005 (+ 932us) server_negotiation.cc:407] Connection header received
0923 18:05:59.899690 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:05:59.899693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:05:59.899776 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:05:59.899898 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:05:59.901504 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.902092 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.903512 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.903837 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.903944 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:59.906719 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:05:59.906724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:05:59.906727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:05:59.906762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:05:59.908482 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:59.909164 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:59.909167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:59.909168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:59.909251 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:05:59.909678 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:05:59.909682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:05:59.909683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:05:59.909892 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:05:59.910000 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:05:59.910612 (+ 612us) server_negotiation.cc:299] Negotiation successful
0923 18:05:59.910740 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:05:59.911978 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:05:59.897271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41181 (local address 127.18.44.193:58505)
0923 18:05:59.897843 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 18:05:59.897889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:05:59.899353 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:05:59.900149 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:05:59.900159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:05:59.900591 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:05:59.901329 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:59.901351 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.902235 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.902239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:59.903386 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:05:59.903397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:05:59.904007 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:05:59.904010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:05:59.904199 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:05:59.904940 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:05:59.904963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:05:59.906505 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:05:59.908629 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:59.908635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:59.908651 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:59.909058 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:59.909344 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:05:59.909346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:05:59.909348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:05:59.909500 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:05:59.910018 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:05:59.910026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:05:59.910337 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 18:05:59.910529 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 18:05:59.910885 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:05:59.913694 24386 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41181
I20260923 18:05:59.913983 24386 heartbeater.cc:464] Registering TS with master...
I20260923 18:05:59.914686 24386 heartbeater.cc:511] Master 127.18.44.254:41181 requested a full tablet report, sending...
I20260923 18:05:59.916736 24195 ts_manager.cc:194] Registered new tserver with Master: 782a5c517a004553b68442ca32c48e08 (127.18.44.193:36255)
I20260923 18:05:59.917980 24195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:58505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:05:59.937124 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:00.100747 24391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:00.101364 24391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:00.101455 24391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:00.106616 24391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:00.106760 24391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:00.106833 24391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:00.114521 24391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:00.116272 24391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:00.117677 24391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:00.132347 24399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:00.132452 24397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:00.132586 24396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:00.132824 24391 server_base.cc:1034] running on GCE node
I20260923 18:06:00.133455 24391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:00.134368 24391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:00.135597 24391 hybrid_clock.cc:648] HybridClock initialized: now 1790186760135553 us; error 62 us; skew 500 ppm
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.139688 24391 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:06:00.141052 24391 webserver.cc:460] Webserver started at http://127.18.44.194:45223/ using document root <none> and password file <none>
I20260923 18:06:00.141825 24391 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:00.141911 24391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:00.142168 24391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:00.144995 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "c66c086f86b74c8aaaa964862b33438c"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "448f3640008b746db888336a0d516c36"
server_key_iv: "6d6d2d3e08c1bf2905f83144a7c38729"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.145779 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "c66c086f86b74c8aaaa964862b33438c"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "448f3640008b746db888336a0d516c36"
server_key_iv: "6d6d2d3e08c1bf2905f83144a7c38729"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.151207 24391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:06:00.155175 24406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.157356 24391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:00.157600 24391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "c66c086f86b74c8aaaa964862b33438c"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "448f3640008b746db888336a0d516c36"
server_key_iv: "6d6d2d3e08c1bf2905f83144a7c38729"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.157804 24391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:00.172178 24391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:00.176786 24391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:00.177042 24391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:00.177881 24391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:00.179330 24391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:00.179407 24391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.179482 24391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:00.179512 24391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.192543 24391 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:45795
I20260923 18:06:00.192585 24519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:45795 every 8 connection(s)
I20260923 18:06:00.194079 24391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:00.197245 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24391
I20260923 18:06:00.197403 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:00.197751 18611 external_mini_cluster.cc:1428] Setting key 6ea51c6a2aa15e4792a21940277b461c
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:00.218750 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.197171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:48977 (local address 127.18.44.254:41181)
0923 18:06:00.197335 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.197339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.199186 (+ 1847us) server_negotiation.cc:407] Connection header received
0923 18:06:00.201207 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.201211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.201300 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.201386 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.203195 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.203952 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.205839 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.206176 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.206396 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.212704 (+ 6308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:00.212710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:00.212715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:00.212779 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:00.214881 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.215988 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.215992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.215994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.216143 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.216778 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.216782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.216784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.217098 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:00.217205 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:00.218254 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0923 18:06:00.218556 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:06:00.219801 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.198131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41181 (local address 127.18.44.194:48977)
0923 18:06:00.198932 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:00.199007 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:00.200835 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:00.201663 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:00.201674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:00.202217 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:00.202953 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:00.202974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.204137 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.204141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:00.205626 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:00.205646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.206553 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.206566 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:00.206879 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.207721 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:00.207746 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:00.212347 (+ 4601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:00.215155 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:00.215166 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:00.215189 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:00.215703 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:00.216327 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:00.216331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:00.216334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:00.216591 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:00.217249 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:00.217258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:00.217722 (+ 464us) client_negotiation.cc:769] Sending connection context
0923 18:06:00.217995 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 18:06:00.218469 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":221,"threads_started":1}
I20260923 18:06:00.222339 24520 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41181
I20260923 18:06:00.222690 24520 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:00.223516 24520 heartbeater.cc:511] Master 127.18.44.254:41181 requested a full tablet report, sending...
I20260923 18:06:00.225821 24195 ts_manager.cc:194] Registered new tserver with Master: c66c086f86b74c8aaaa964862b33438c (127.18.44.194:45795)
I20260923 18:06:00.226545 24195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:48977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:06:00.237656 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:00.363214 24525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:00.363608 24525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:00.363682 24525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:00.368162 24525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:00.368252 24525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:00.368338 24525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:00.374034 24525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:00.375312 24525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:00.376407 24525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:00.388453 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:00.388592 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:00.388588 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:00.389127 24525 server_base.cc:1034] running on GCE node
I20260923 18:06:00.389752 24525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:00.390642 24525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:00.392022 24525 hybrid_clock.cc:648] HybridClock initialized: now 1790186760391970 us; error 89 us; skew 500 ppm
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.396401 24525 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:06:00.397905 24525 webserver.cc:460] Webserver started at http://127.18.44.195:33057/ using document root <none> and password file <none>
I20260923 18:06:00.398674 24525 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:00.398762 24525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:00.399070 24525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:00.402098 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "6a8650724cd548239cc3464caa379929"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "d52bb789880f386caf5b61011b884092"
server_key_iv: "f99f95406517ba15c00aa1d741ea61fe"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.402959 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "6a8650724cd548239cc3464caa379929"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "d52bb789880f386caf5b61011b884092"
server_key_iv: "f99f95406517ba15c00aa1d741ea61fe"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.407959 24525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:06:00.411731 24540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.413848 24525 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:00.414034 24525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "6a8650724cd548239cc3464caa379929"
format_stamp: "Formatted at 2026-09-23 18:06:00 on dist-test-slave-rvjj"
server_key: "d52bb789880f386caf5b61011b884092"
server_key_iv: "f99f95406517ba15c00aa1d741ea61fe"
server_key_version: "encryptionkey@0"
I20260923 18:06:00.414201 24525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:00.432157 24525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:00.436967 24525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:00.437183 24525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:00.438035 24525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:00.439167 24525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:00.439224 24525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.439313 24525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:00.439354 24525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:00.451494 24653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:35977 every 8 connection(s)
I20260923 18:06:00.451581 24525 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:35977
I20260923 18:06:00.453127 24525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:00.457559 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24525
I20260923 18:06:00.457736 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:00.458425 18611 external_mini_cluster.cc:1428] Setting key ff019da3a225124685714b2b31a26ab8
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:00.472743 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.455476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:41937 (local address 127.18.44.254:41181)
0923 18:06:00.455725 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.455730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.457528 (+ 1798us) server_negotiation.cc:407] Connection header received
0923 18:06:00.459702 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.459706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.459779 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.459869 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.461360 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.462213 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.463881 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.464290 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.464413 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.468105 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:00.468110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:00.468113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:00.468156 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:00.469991 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.470914 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.470919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.470921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.471023 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.471480 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.471483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.471484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.471721 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:00.471823 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:00.472397 (+ 574us) server_negotiation.cc:299] Negotiation successful
0923 18:06:00.472557 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 18:06:00.473742 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.456225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41181 (local address 127.18.44.195:41937)
0923 18:06:00.457107 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:00.457332 (+ 225us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:00.459370 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:00.460111 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:00.460121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:00.460561 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:00.461157 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:00.461183 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.462407 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.462410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:00.463721 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:00.463744 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.464486 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.464490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:00.464735 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.465478 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:00.465503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:00.467773 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:00.470210 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:00.470226 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:00.470244 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:00.470693 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:00.471125 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:00.471128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:00.471130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:00.471371 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:00.471839 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:00.471847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:00.472137 (+ 290us) client_negotiation.cc:769] Sending connection context
0923 18:06:00.472318 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 18:06:00.472703 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":237,"threads_started":1}
I20260923 18:06:00.475191 24654 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41181
I20260923 18:06:00.475526 24654 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:00.476121 24654 heartbeater.cc:511] Master 127.18.44.254:41181 requested a full tablet report, sending...
I20260923 18:06:00.477738 24195 ts_manager.cc:194] Registered new tserver with Master: 6a8650724cd548239cc3464caa379929 (127.18.44.195:35977)
I20260923 18:06:00.478458 24195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:41937
I20260923 18:06:00.489813 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/test-admin.keytab.
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.515686 18611 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 18:06:00.520749 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:00.517507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.18.44.193:36255)
0923 18:06:00.518002 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.518012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.518131 (+ 119us) server_negotiation.cc:407] Connection header received
0923 18:06:00.518273 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.518284 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.518699 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.518899 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.520515 (+ 1616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47378: BlockingRecv error: recv got EOF from 127.0.0.1:47378 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":191,"threads_started":1}
W20260923 18:06:00.520995 18611 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc' of type FILE
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.522552 18611 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.524506 18611 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:06:00.539788 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.525840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47380 (local address 127.18.44.193:36255)
0923 18:06:00.525987 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.525991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.526177 (+ 186us) server_negotiation.cc:407] Connection header received
0923 18:06:00.526416 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.526421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.526497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.526594 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.527554 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.528576 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.530382 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.530864 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.530996 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.534872 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:00.534889 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:00.534905 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:00.534943 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:00.537255 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.537829 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.537835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.537836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.537949 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.538490 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.538493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.538494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.538888 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:00.539086 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:00.539421 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 18:06:00.539595 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 18:06:00.545248 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:00.543887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.18.44.193:36255)
0923 18:06:00.544092 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.544096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.544185 (+ 89us) server_negotiation.cc:407] Connection header received
0923 18:06:00.544317 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.544319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.544367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.544447 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.545131 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47390: BlockingRecv error: recv got EOF from 127.0.0.1:47390 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260923 18:06:00.545311 18611 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc' of type FILE
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.546761 18611 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.548024 18611 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:06:00.561869 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.549511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.18.44.193:36255)
0923 18:06:00.549695 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.549699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.549861 (+ 162us) server_negotiation.cc:407] Connection header received
0923 18:06:00.549992 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.549994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.550045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.550135 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.550971 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.551824 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.553110 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.553378 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.553500 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.557292 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:00.557297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:00.557299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:00.557349 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:00.559544 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.560180 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.560184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.560185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.560289 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.560764 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.560768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.560769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.561036 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:00.561252 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:00.561527 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 18:06:00.561678 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260923 18:06:00.566766 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:00.565304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.18.44.193:36255)
0923 18:06:00.565518 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.565526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.565685 (+ 159us) server_negotiation.cc:407] Connection header received
0923 18:06:00.565874 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.565878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.565934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.566013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.566615 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47418: BlockingRecv error: recv got EOF from 127.0.0.1:47418 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260923 18:06:00.567412 18611 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790186727112038-18611-0/krb5kdc/krb5cc' of type FILE
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.569185 18611 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:00.570590 18611 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:06:00.586354 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:00.572699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.18.44.193:36255)
0923 18:06:00.572883 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:00.572888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:00.572979 (+ 91us) server_negotiation.cc:407] Connection header received
0923 18:06:00.573228 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:00.573232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:00.573288 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:00.573713 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:00.574491 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.575071 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.576278 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:00.576814 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:00.576985 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:00.581079 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:00.581084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:00.581086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:00.581137 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:00.583834 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.584504 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.584508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.584509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.584615 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:00.584898 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:00.584902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:00.584904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:00.585146 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:00.585418 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:00.585985 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 18:06:00.586154 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 18:06:00.588331 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24257
I20260923 18:06:00.594890 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24391
I20260923 18:06:00.604336 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24525
I20260923 18:06:00.610931 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24163
2026-09-23T18:06:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2500 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24674](info): set up 2 sockets
Sep 23 18:06:00 dist-test-slave-rvjj krb5kdc[24674](info): commencing operation
krb5kdc: starting...
W20260923 18:06:01.657523 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:01 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab.
Entry for principal oryx/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab.
I20260923 18:06:01.797742 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44433
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab
--principal=oryx/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:44433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:01.939085 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:01.939450 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:01.939495 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:01.943897 24687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:01.943990 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:01.944008 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:01.944025 24687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:01.944039 24687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:01.950316 24687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:44433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab
--principal=oryx/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:01.951638 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:01.952877 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:01.963423 24693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:01.963425 24692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:01.963591 24687 server_base.cc:1034] running on GCE node
W20260923 18:06:01.963505 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:01.964389 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:01.965652 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:01.966908 24687 hybrid_clock.cc:648] HybridClock initialized: now 1790186761966854 us; error 72 us; skew 500 ppm
Sep 23 18:06:01 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:01.970877 24687 init.cc:377] Logged in from keytab as oryx/127.18.44.254@KRBTEST.COM (short username oryx)
I20260923 18:06:01.972234 24687 webserver.cc:460] Webserver started at http://127.18.44.254:46643/ using document root <none> and password file <none>
I20260923 18:06:01.972980 24687 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:01.973069 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:01.973289 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:01.975656 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "d0891315db754d67be66589360f29de3"
format_stamp: "Formatted at 2026-09-23 18:06:01 on dist-test-slave-rvjj"
server_key: "120eea27e858c1ab736e27b1ed14675b"
server_key_iv: "e0572c1fa8562f734dab391737057594"
server_key_version: "encryptionkey@0"
I20260923 18:06:01.976279 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "d0891315db754d67be66589360f29de3"
format_stamp: "Formatted at 2026-09-23 18:06:01 on dist-test-slave-rvjj"
server_key: "120eea27e858c1ab736e27b1ed14675b"
server_key_iv: "e0572c1fa8562f734dab391737057594"
server_key_version: "encryptionkey@0"
I20260923 18:06:01.980512 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:01.983856 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:01.985842 24687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:01.986071 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "d0891315db754d67be66589360f29de3"
format_stamp: "Formatted at 2026-09-23 18:06:01 on dist-test-slave-rvjj"
server_key: "120eea27e858c1ab736e27b1ed14675b"
server_key_iv: "e0572c1fa8562f734dab391737057594"
server_key_version: "encryptionkey@0"
I20260923 18:06:01.986359 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:02.010844 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:02.018675 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:02.018975 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:02.032815 24687 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:44433
I20260923 18:06:02.032972 24754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:44433 every 8 connection(s)
I20260923 18:06:02.038283 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:02.042892 24755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:02.047159 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24687
I20260923 18:06:02.047411 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:02.047802 18611 external_mini_cluster.cc:1428] Setting key 3824c00dc272eb8159440d9bc73e4d71
I20260923 18:06:02.050781 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Bootstrap starting.
I20260923 18:06:02.053509 24755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:02.055203 24755 log.cc:824] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:02.057803 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: No bootstrap required, opened a new log
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:02.063663 24755 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } }
I20260923 18:06:02.063970 24755 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:02.064069 24755 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0891315db754d67be66589360f29de3, State: Initialized, Role: FOLLOWER
I20260923 18:06:02.064675 24755 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } }
I20260923 18:06:02.064862 24755 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:02.064934 24755 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:02.065032 24755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:02.066094 24755 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } }
I20260923 18:06:02.066486 24755 leader_election.cc:302] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0891315db754d67be66589360f29de3; no voters:
I20260923 18:06:02.066864 24755 leader_election.cc:288] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:02.067121 24760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:02.067613 24760 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 LEADER]: Becoming Leader. State: Replica: d0891315db754d67be66589360f29de3, State: Running, Role: LEADER
I20260923 18:06:02.068125 24760 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } }
I20260923 18:06:02.068392 24755 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:02.070612 24762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0891315db754d67be66589360f29de3. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } }
I20260923 18:06:02.070840 24762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:02.071292 24761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } }
I20260923 18:06:02.071437 24761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:02.071812 24771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:02.073638 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.050237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.18.44.254:44433)
0923 18:06:02.051391 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:02.051416 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:02.051460 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:06:02.052319 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:02.052340 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:02.053074 (+ 734us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:02.053693 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:02.055928 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.057682 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.059325 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.060120 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.060347 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:02.064861 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:02.064872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:02.064905 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:02.065066 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:02.067552 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.068430 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.068437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.068446 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.068658 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.069202 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.069206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.069208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.069949 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:02.070208 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:02.071729 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0923 18:06:02.072243 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":372,"threads_started":1}
I20260923 18:06:02.077135 24771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:02.088645 24771 catalog_manager.cc:1347] Generated new cluster ID: ba34ba7ebde84ca49875e1e917af0539
I20260923 18:06:02.088763 24771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:02.101322 24771 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:02.102609 24771 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:02.117357 24771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Generated new TSK 0
I20260923 18:06:02.118244 24771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
I20260923 18:06:02.160490 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:02.289983 24781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:02.290282 24781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:02.290340 24781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:02.294570 24781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:02.294658 24781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:02.294744 24781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:02.300323 24781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:02.301436 24781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:02.302448 24781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:02.313504 24789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.313807 24786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.313504 24787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:02.314033 24781 server_base.cc:1034] running on GCE node
I20260923 18:06:02.314697 24781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:02.315516 24781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:02.316922 24781 hybrid_clock.cc:648] HybridClock initialized: now 1790186762316895 us; error 66 us; skew 500 ppm
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:02.320514 24781 init.cc:377] Logged in from keytab as oryx/127.18.44.193@KRBTEST.COM (short username oryx)
I20260923 18:06:02.321712 24781 webserver.cc:460] Webserver started at http://127.18.44.193:46715/ using document root <none> and password file <none>
I20260923 18:06:02.322328 24781 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:02.322403 24781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:02.322606 24781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:02.324985 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "c3cd43e195e24ea2af9092dcf35ed533"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "febb48fe28ef24cc3a944707a5267c50"
server_key_iv: "785cb29cb1300b4fd91eaebd75ca0989"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.325609 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "c3cd43e195e24ea2af9092dcf35ed533"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "febb48fe28ef24cc3a944707a5267c50"
server_key_iv: "785cb29cb1300b4fd91eaebd75ca0989"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.330618 24781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:06:02.334060 24796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.336236 24781 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:06:02.336413 24781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "c3cd43e195e24ea2af9092dcf35ed533"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "febb48fe28ef24cc3a944707a5267c50"
server_key_iv: "785cb29cb1300b4fd91eaebd75ca0989"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.336545 24781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:02.357416 24781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:02.361647 24781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:02.361878 24781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:02.362520 24781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:02.363572 24781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:02.363631 24781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.363698 24781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:02.363724 24781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.374516 24781 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:46483
I20260923 18:06:02.374621 24909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:46483 every 8 connection(s)
I20260923 18:06:02.375628 24781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:02.380998 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24781
I20260923 18:06:02.381138 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:02.381455 18611 external_mini_cluster.cc:1428] Setting key d49162d402c50ee610be6d2d8f0c567a
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.194@KRBTEST.COM" created.
I20260923 18:06:02.397969 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.378397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35189 (local address 127.18.44.254:44433)
0923 18:06:02.378610 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:02.378614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:02.379876 (+ 1262us) server_negotiation.cc:407] Connection header received
0923 18:06:02.381980 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:02.381985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:02.382059 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:02.382152 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:02.384175 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.384966 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.386947 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.387369 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.387513 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:02.391897 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:02.391902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:02.391904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:02.391943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:02.393888 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.395404 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.395408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.395410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.395533 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.396245 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.396248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.396249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.396497 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:02.396589 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:02.397621 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0923 18:06:02.397821 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 18:06:02.399787 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.378741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:35189)
0923 18:06:02.379519 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:02.379639 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:02.381630 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:02.382468 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:02.382479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:02.383144 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:02.383930 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:02.383949 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.385178 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.385183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:02.386710 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:02.386725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.387701 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.387706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:02.387982 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:02.388766 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:02.388797 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:02.391383 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:02.394188 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:02.394197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:02.394218 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:02.395115 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:02.395781 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:02.395784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:02.395786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:02.396071 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:02.396678 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:02.396686 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:02.397046 (+ 360us) client_negotiation.cc:769] Sending connection context
0923 18:06:02.397544 (+ 498us) client_negotiation.cc:240] Negotiation successful
0923 18:06:02.398162 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":139,"threads_started":1}
I20260923 18:06:02.402299 24910 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:02.402822 24910 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:02.403795 24910 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
I20260923 18:06:02.406531 24719 ts_manager.cc:194] Registered new tserver with Master: c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193:46483)
I20260923 18:06:02.408304 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:35189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
I20260923 18:06:02.426337 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:02.584000 24915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:02.584424 24915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:02.584511 24915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:02.590271 24915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:02.590404 24915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:02.590592 24915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:02.598874 24915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:02.600548 24915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:02.601939 24915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:06:02.616850 24915 server_base.cc:1034] running on GCE node
W20260923 18:06:02.616771 24921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.616825 24920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.616771 24923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:02.617791 24915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:02.618851 24915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:02.620162 24915 hybrid_clock.cc:648] HybridClock initialized: now 1790186762620127 us; error 59 us; skew 500 ppm
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:02.624626 24915 init.cc:377] Logged in from keytab as oryx/127.18.44.194@KRBTEST.COM (short username oryx)
I20260923 18:06:02.626441 24915 webserver.cc:460] Webserver started at http://127.18.44.194:35337/ using document root <none> and password file <none>
I20260923 18:06:02.627992 24915 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:02.628155 24915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:02.628683 24915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:02.632555 24915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "bc9aab5e9576496e809caf334432579b"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "403327d65686d4835787e05668e1168c"
server_key_iv: "00be751ce9a6c83eebfde63681e52161"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.633816 24915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "bc9aab5e9576496e809caf334432579b"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "403327d65686d4835787e05668e1168c"
server_key_iv: "00be751ce9a6c83eebfde63681e52161"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.640357 24915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:06:02.644811 24930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.647213 24915 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:06:02.647569 24915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "bc9aab5e9576496e809caf334432579b"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "403327d65686d4835787e05668e1168c"
server_key_iv: "00be751ce9a6c83eebfde63681e52161"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.647751 24915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:02.666256 24915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:02.671744 24915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:02.672020 24915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:02.672871 24915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:02.674098 24915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:02.674160 24915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.674226 24915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:02.674255 24915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.686894 24915 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:45397
I20260923 18:06:02.686952 25043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:45397 every 8 connection(s)
I20260923 18:06:02.688225 24915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:02.697052 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 24915
I20260923 18:06:02.697221 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:02.697597 18611 external_mini_cluster.cc:1428] Setting key 6a190dfc7cacfea97dadca7c42cb3ca6
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:02.709816 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.690981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:34001 (local address 127.18.44.254:44433)
0923 18:06:02.691340 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:02.691346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:02.692894 (+ 1548us) server_negotiation.cc:407] Connection header received
0923 18:06:02.694825 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:02.694830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:02.694920 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:02.695058 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:02.696791 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.697686 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.699391 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.699825 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.699978 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:02.703521 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:02.703527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:02.703530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:02.703576 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:02.705567 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.706772 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.706777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.706778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.706907 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:02.707625 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:02.707629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:02.707630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:02.707873 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:02.707979 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:02.709495 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0923 18:06:02.709672 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
WARNING: no policy specified for oryx/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:06:02.711613 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.691660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:34001)
0923 18:06:02.692594 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:02.692685 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:02.694426 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:02.695396 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:02.695407 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:02.695925 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:02.696525 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:02.696544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.697880 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.697883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:02.699157 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:02.699172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:02.700054 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:02.700058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:02.700267 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:02.700946 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:02.700965 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:02.702981 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:02.705860 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:02.705873 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:02.705904 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:02.706532 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:02.707069 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:02.707074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:02.707076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:02.707439 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:02.708028 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:02.708040 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:02.708872 (+ 832us) client_negotiation.cc:769] Sending connection context
0923 18:06:02.709324 (+ 452us) client_negotiation.cc:240] Negotiation successful
0923 18:06:02.709922 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":251,"threads_started":1}
I20260923 18:06:02.713944 25044 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:02.714308 25044 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:02.715088 25044 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.195@KRBTEST.COM" created.
I20260923 18:06:02.717046 24719 ts_manager.cc:194] Registered new tserver with Master: bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:02.717888 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
I20260923 18:06:02.741298 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:02.891511 25049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:02.891974 25049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:02.892052 25049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:02.897397 25049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:02.897533 25049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:02.897634 25049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:02.905905 25049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:02.907526 25049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:02.908906 25049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:02.922231 25054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.922313 25057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:02.922338 25055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:02.922462 25049 server_base.cc:1034] running on GCE node
I20260923 18:06:02.923635 25049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:02.924940 25049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:02.926262 25049 hybrid_clock.cc:648] HybridClock initialized: now 1790186762926234 us; error 45 us; skew 500 ppm
Sep 23 18:06:02 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:02.930371 25049 init.cc:377] Logged in from keytab as oryx/127.18.44.195@KRBTEST.COM (short username oryx)
I20260923 18:06:02.932150 25049 webserver.cc:460] Webserver started at http://127.18.44.195:37847/ using document root <none> and password file <none>
I20260923 18:06:02.933310 25049 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:02.933631 25049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:02.934015 25049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:02.937107 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "7133a7fcb4fe45a08470cc277e8bca73"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "01ed7fb37c83dbd011c504b074b1707c"
server_key_iv: "940d651cff646177363ef335b81142eb"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.938190 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "7133a7fcb4fe45a08470cc277e8bca73"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "01ed7fb37c83dbd011c504b074b1707c"
server_key_iv: "940d651cff646177363ef335b81142eb"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.945410 25049 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 18:06:02.949561 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.951241 25049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:06:02.951448 25049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "7133a7fcb4fe45a08470cc277e8bca73"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "01ed7fb37c83dbd011c504b074b1707c"
server_key_iv: "940d651cff646177363ef335b81142eb"
server_key_version: "encryptionkey@0"
I20260923 18:06:02.951586 25049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:02.974488 25049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:02.978838 25049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:02.979043 25049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:02.979825 25049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:02.980922 25049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:02.980975 25049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.981029 25049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:02.981050 25049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:02.992496 25049 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46033
I20260923 18:06:02.992514 25177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46033 every 8 connection(s)
I20260923 18:06:02.993609 25049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:03.003973 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25049
I20260923 18:06:03.004155 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:03.004536 18611 external_mini_cluster.cc:1428] Setting key 2bc7559956a9f1fa3bef2e9a5e9b5a56
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186762, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:03.013878 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.996499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50883 (local address 127.18.44.254:44433)
0923 18:06:02.996745 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:02.996751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:02.998425 (+ 1674us) server_negotiation.cc:407] Connection header received
0923 18:06:02.999704 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:02.999709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:02.999789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:02.999911 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.001480 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.002123 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.003758 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.004140 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.004292 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.008134 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.008138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.008140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.008180 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.010265 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.011404 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.011409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.011410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.011548 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.012124 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.012127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.012128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.012383 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.012488 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.013541 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.013715 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 18:06:03.015158 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:02.997071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:50883)
0923 18:06:02.998142 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:02.998244 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:02.999375 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.000189 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.000198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.000630 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.001214 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.001230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.002372 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.002376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.003565 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.003579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.004393 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.004398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.004604 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.005522 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.005544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.007905 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.010479 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.010486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.010502 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.011136 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.011684 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.011687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.011689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.011927 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.012576 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.012584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.013126 (+ 542us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.013402 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.013912 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":201,"threads_started":1}
I20260923 18:06:03.017405 25178 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:03.017802 25178 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:03.018482 25178 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
I20260923 18:06:03.019907 24719 ts_manager.cc:194] Registered new tserver with Master: 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033)
I20260923 18:06:03.020527 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:50883
I20260923 18:06:03.020812 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.44.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.253.keytab.
Entry for principal oryx/127.18.44.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.253.keytab.
I20260923 18:06:03.059672 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:44899
--webserver_interface=127.18.44.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.253.keytab
--principal=oryx/127.18.44.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:44433,127.18.44.253:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:03.213863 25183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:03.214315 25183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:03.214385 25183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:03.219547 25183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:03.219666 25183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:03.219694 25183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:03.219718 25183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:03.219738 25183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:03.228471 25183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:44433,127.18.44.253:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.253.keytab
--principal=oryx/127.18.44.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.44.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
W20260923 18:06:03.228863 25183 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.44.254:44433,127.18.44.253:44899'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 18:06:03.230156 25183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:03.231633 25183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:03.243009 25188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.243009 25189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.243126 25191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.243582 25183 server_base.cc:1034] running on GCE node
I20260923 18:06:03.244469 25183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:03.246140 25183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:03.247394 25183 hybrid_clock.cc:648] HybridClock initialized: now 1790186763247353 us; error 67 us; skew 500 ppm
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:03.251765 25183 init.cc:377] Logged in from keytab as oryx/127.18.44.253@KRBTEST.COM (short username oryx)
I20260923 18:06:03.253307 25183 webserver.cc:460] Webserver started at http://127.18.44.253:45431/ using document root <none> and password file <none>
I20260923 18:06:03.254127 25183 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.254222 25183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.254489 25183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:03.257979 25183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/instance:
uuid: "88baa141f23e40caa57798777380a6b5"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "be8205207631f35d4aec9d91f4196fbc"
server_key_iv: "7fa950752c92d10ecb4553ff06635d4d"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.258865 25183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal/instance:
uuid: "88baa141f23e40caa57798777380a6b5"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "be8205207631f35d4aec9d91f4196fbc"
server_key_iv: "7fa950752c92d10ecb4553ff06635d4d"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.265183 25183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260923 18:06:03.269337 25198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.271500 25183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:03.271692 25183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "88baa141f23e40caa57798777380a6b5"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "be8205207631f35d4aec9d91f4196fbc"
server_key_iv: "7fa950752c92d10ecb4553ff06635d4d"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.271864 25183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.294646 25183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:03.298080 25183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:03.298265 25183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:03.306393 25183 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.253:44899
I20260923 18:06:03.306813 25250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.253:44899 every 8 connection(s)
I20260923 18:06:03.307631 25183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
I20260923 18:06:03.309660 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25183
I20260923 18:06:03.312531 25251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:03.318336 25251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.253@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:03.328434 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.312121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.18.44.253:44899)
0923 18:06:03.312900 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.312923 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.312983 (+ 60us) server_negotiation.cc:407] Connection header received
0923 18:06:03.313786 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.313810 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.314258 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.314684 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.315647 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.317021 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.318221 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.318974 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.319161 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.322442 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.322448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.322458 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.322489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.324559 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.325083 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.325090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.325095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.325226 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.325523 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.325525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.325527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.326159 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.326411 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.326761 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.327110 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":198,"threads_started":1}
I20260923 18:06:03.334029 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.320954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.18.44.254:44433)
0923 18:06:03.321196 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.321202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.321622 (+ 420us) server_negotiation.cc:407] Connection header received
0923 18:06:03.321736 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.321739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.321814 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.321898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.322814 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.323717 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.324954 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.325210 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.325317 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.328501 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.328506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.328509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.328545 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.331306 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.332011 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.332015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.332018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.332121 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.332500 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.332506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.332508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.332937 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.333055 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.333655 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.333821 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 18:06:03.334281 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.320888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38038)
0923 18:06:03.321426 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.321487 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.321636 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.321994 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.322001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.322374 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.322518 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.322524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.323927 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.323932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.324781 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.324797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.325342 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.325346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.325512 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.326316 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.326330 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.328246 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.331473 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.331483 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.331492 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.331879 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.332217 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.332220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.332222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.332315 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.333113 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.333120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.333291 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.333484 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.333863 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260923 18:06:03.336364 25251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.253@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
I20260923 18:06:03.347508 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.337073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46802)
0923 18:06:03.337281 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.337300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.337425 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.337641 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.337644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.337919 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.338074 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.338082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.339517 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.339520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.340211 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.340223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.340925 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.340929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.341042 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.341785 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.341801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.343480 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.345428 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.345431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.345432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.345638 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.345987 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.345989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.345991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.346073 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.346713 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.346716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.346879 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.347021 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.347211 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 18:06:03.347522 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.337173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.18.44.253:44899)
0923 18:06:03.337281 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.337286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.337367 (+ 81us) server_negotiation.cc:407] Connection header received
0923 18:06:03.337519 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.337522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.337556 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.337615 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.338284 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.339339 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.340397 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.340718 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.340831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.343769 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.343773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.343775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.343814 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.345270 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.345792 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.345795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.345796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.345874 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.346325 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.346328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.346330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.346547 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.346639 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.347135 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.347356 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:06:03.352123 25251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Bootstrap starting.
I20260923 18:06:03.355873 25251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:03.357091 25251 log.cc:824] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:03.361100 25251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: No bootstrap required, opened a new log
I20260923 18:06:03.365341 25251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } }
I20260923 18:06:03.365723 25251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:03.365782 25251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88baa141f23e40caa57798777380a6b5, State: Initialized, Role: FOLLOWER
I20260923 18:06:03.366385 25251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } }
I20260923 18:06:03.367182 25257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } } }
I20260923 18:06:03.367440 25257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:03.368245 25251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:06:03.376420 25268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:06:03.376601 25268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:06:03.378964 25183 master_runner.cc:428] Detected that this master 88baa141f23e40caa57798777380a6b5 is joining an existing cluster
I20260923 18:06:03.379060 25183 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.44.253:44899
I20260923 18:06:03.391887 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.381477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.18.44.253:44899)
0923 18:06:03.381595 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.381599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.381899 (+ 300us) server_negotiation.cc:407] Connection header received
0923 18:06:03.382036 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.382039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.382077 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.382147 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.382715 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.383690 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.385117 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.385471 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.385554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.387976 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.387980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.387983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.388021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.389870 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390405 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.390409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.390411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.390486 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390752 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.390755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.390757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.391018 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.391088 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.391438 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.391592 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:06:03.392194 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.381824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.18.44.254:44433)
0923 18:06:03.381969 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.381975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.382293 (+ 318us) server_negotiation.cc:407] Connection header received
0923 18:06:03.382592 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.382594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.382651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.382727 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.383446 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.384201 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.385391 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.385714 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.385793 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.387804 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.387808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.387811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.387842 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.390009 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390648 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.390652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.390654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.390750 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.391094 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.391097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.391099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.391391 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.391467 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.391867 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.392013 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:06:03.392838 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.381633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38048)
0923 18:06:03.382128 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.382149 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.382366 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.382744 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.382747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.383132 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.383293 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.383301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.384355 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.384358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.385249 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.385260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.386000 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.386004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.386106 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.386927 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.386943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.387604 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.390201 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.390206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.390507 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.390871 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.390875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.390969 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.391543 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.391546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.391674 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.392447 (+ 773us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.392637 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":85,"threads_started":1}
I20260923 18:06:03.393332 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.381385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46806)
0923 18:06:03.381818 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.381834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.381964 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.382182 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.382184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.382421 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.382549 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.382560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.383880 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.383884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.384923 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.384938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.385772 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.385775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.385871 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.387097 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.387116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.387778 (+ 662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.390045 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.390051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.390309 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.390589 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.390592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.390594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.390674 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.391145 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.391148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.391226 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.393026 (+ 1800us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.393176 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for oryx/127.18.44.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.252@KRBTEST.COM" created.
I20260923 18:06:03.412848 24910 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.254@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
I20260923 18:06:03.419423 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.399592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.18.44.253:44899)
0923 18:06:03.399761 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.399766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.401254 (+ 1488us) server_negotiation.cc:407] Connection header received
0923 18:06:03.401541 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.401546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.401640 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.401785 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.402708 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.403790 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.406364 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.406867 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.407009 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.414639 (+ 7630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.414646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.414649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.414719 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.416784 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.417711 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.417715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.417717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.417819 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.418190 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.418193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.418195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.418418 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.418507 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.419005 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.419141 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:06:03.419451 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.399469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46810)
0923 18:06:03.401043 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.401114 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.401325 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.401864 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.401872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.402381 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.402532 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.402540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.404055 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.404060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.405795 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.405827 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.407367 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.407372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.407609 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.408531 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.408550 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.414342 (+ 5792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.416945 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.416962 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.416971 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.417505 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.417996 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.417999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.418000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.418107 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.418527 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.418536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.418688 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.418940 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.419195 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":101,"threads_started":1}
I20260923 18:06:03.421171 24719 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.44.253:44899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "88baa141f23e40caa57798777380a6b5"
member_type: NON_VOTER
last_known_addr {
host: "127.18.44.253"
port: 44899
}
attrs {
promote: true
}
}
dest_uuid: "d0891315db754d67be66589360f29de3"
cas_config_opid_index: -1
I20260923 18:06:03.421954 24719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } }
I20260923 18:06:03.424185 25281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88baa141f23e40caa57798777380a6b5 (127.18.44.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } }
I20260923 18:06:03.425552 25225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:03.427320 25225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.252.keytab.
Entry for principal oryx/127.18.44.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.252.keytab.
I20260923 18:06:03.428339 25280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:03.428553 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:43589
--webserver_interface=127.18.44.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.252.keytab
--principal=oryx/127.18.44.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:06:03.428627 25281 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.44.253:44899
I20260923 18:06:03.429346 25281 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } }
I20260923 18:06:03.429507 25281 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:03.429504 25183 master.cc:537] Master@127.18.44.253:44899 shutting down...
W20260923 18:06:03.433081 24703 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 18:06:03.434428 24703 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 -> Peer 88baa141f23e40caa57798777380a6b5 (127.18.44.253:44899): Couldn't send request to peer 88baa141f23e40caa57798777380a6b5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:06:03.437309 25183 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:03.437712 25183 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:03.438071 25183 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: stopping tablet replica
I20260923 18:06:03.453869 25183 master.cc:552] Master@127.18.44.253:44899 shutdown complete.
I20260923 18:06:03.453938 25183 master_runner.cc:305] Clearing existing system tablet
I20260923 18:06:03.454186 25183 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.454239 25183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.455681 25183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 18:06:03.456555 25288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.456815 25183 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.456946 25183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.456981 25183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "88baa141f23e40caa57798777380a6b5"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "be8205207631f35d4aec9d91f4196fbc"
server_key_iv: "7fa950752c92d10ecb4553ff06635d4d"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.458905 25183 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:03.461781 25183 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 18:06:03.461896 25183 log.cc:1196] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 18:06:03.462160 25183 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Deleting consensus metadata
I20260923 18:06:03.462337 25183 master_runner.cc:315] Copying system tablet from 127.18.44.254:44433
I20260923 18:06:03.463580 25183 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: tablet copy: Beginning tablet copy session from remote peer at address 127.18.44.254:44433
I20260923 18:06:03.472446 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.464144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38062)
0923 18:06:03.464389 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.464404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.464500 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.465250 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.465253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.465438 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.465545 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.465550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.466370 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.466374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.466969 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.466977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.467503 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.467506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.467585 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.468119 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.468131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.468509 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.470404 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.470408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.470411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.470854 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.471355 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.471358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.471361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.471455 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.471861 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.471864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.471965 (+ 101us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.472053 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.472238 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
I20260923 18:06:03.472438 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.464635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.18.44.254:44433)
0923 18:06:03.464956 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.464960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.464975 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:06:03.465015 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.465017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.465056 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.465139 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.465730 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.466264 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.467079 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.467395 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.467488 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.468649 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.468653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.468656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.468682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.470286 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.471014 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.471018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.471020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.471217 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.471547 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.471551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.471553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.471765 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.471856 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.472132 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.472257 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260923 18:06:03.473354 24739 tablet_copy_service.cc:140] P d0891315db754d67be66589360f29de3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 88baa141f23e40caa57798777380a6b5 ({username='oryx', principal='oryx/127.18.44.253@KRBTEST.COM'} at 127.0.0.1:38062)
I20260923 18:06:03.473699 24739 tablet_copy_service.cc:161] P d0891315db754d67be66589360f29de3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 88baa141f23e40caa57798777380a6b5 at {username='oryx', principal='oryx/127.18.44.253@KRBTEST.COM'} at 127.0.0.1:38062: session id = 88baa141f23e40caa57798777380a6b5-00000000000000000000000000000000
I20260923 18:06:03.475744 24739 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 18:06:03.477692 25183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 18:06:03.480005 25183 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: tablet copy: Starting download of 0 data blocks...
I20260923 18:06:03.480199 25183 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: tablet copy: Starting download of 1 WAL segments...
I20260923 18:06:03.482154 25183 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 18:06:03.484148 24739 tablet_copy_service.cc:342] P d0891315db754d67be66589360f29de3: Request end of tablet copy session 88baa141f23e40caa57798777380a6b5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.44.253@KRBTEST.COM'} at 127.0.0.1:38062
I20260923 18:06:03.484351 24739 tablet_copy_service.cc:433] P d0891315db754d67be66589360f29de3: ending tablet copy session 88baa141f23e40caa57798777380a6b5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 88baa141f23e40caa57798777380a6b5
I20260923 18:06:03.492204 25183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:03.494288 25298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.494520 25295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.494809 25183 server_base.cc:1034] running on GCE node
W20260923 18:06:03.494944 25296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.495204 25183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:03.495597 25183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:03.503029 25183 hybrid_clock.cc:648] HybridClock initialized: now 1790186763503014 us; error 46 us; skew 500 ppm
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:03.504709 25183 init.cc:377] Logged in from keytab as oryx/127.18.44.253@KRBTEST.COM (short username oryx)
I20260923 18:06:03.505260 25183 webserver.cc:460] Webserver started at http://127.18.44.253:37115/ using document root <none> and password file <none>
I20260923 18:06:03.505450 25183 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.505496 25183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.506630 25183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.507577 25305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.507807 25183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.507880 25183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "88baa141f23e40caa57798777380a6b5"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "be8205207631f35d4aec9d91f4196fbc"
server_key_iv: "7fa950752c92d10ecb4553ff06635d4d"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.507968 25183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.533854 25183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:03.535857 25183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:03.540740 25183 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.253:44899
I20260923 18:06:03.541389 25183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
I20260923 18:06:03.543164 25358 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:06:03.544087 25358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Bootstrap starting.
I20260923 18:06:03.551373 25357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.253:44899 every 8 connection(s)
I20260923 18:06:03.554699 25358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:06:03.555444 25358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: Bootstrap complete.
I20260923 18:06:03.556156 25358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } }
I20260923 18:06:03.556288 25358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88baa141f23e40caa57798777380a6b5, State: Initialized, Role: LEARNER
I20260923 18:06:03.556402 25358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } }
I20260923 18:06:03.556952 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } }
I20260923 18:06:03.557183 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:06:03.557199 25358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:03.560802 25372 catalog_manager.cc:1259] Loaded cluster ID: ba34ba7ebde84ca49875e1e917af0539
I20260923 18:06:03.560971 25372 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: loading cluster ID for follower catalog manager: success
I20260923 18:06:03.564248 25372 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: acquiring CA information for follower catalog manager: success
I20260923 18:06:03.565791 25372 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 18:06:03.644181 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:03.644630 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:03.644758 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:03.651208 25285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:03.651526 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:03.651562 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:03.651587 25285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:03.651611 25285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:03.660050 25285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.252.keytab
--principal=oryx/127.18.44.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.44.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:03.661690 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:03.663426 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:03.675185 25377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.675244 25378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.675215 25380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.675746 25285 server_base.cc:1034] running on GCE node
I20260923 18:06:03.676431 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:03.678010 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:03.679427 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790186763679397 us; error 46 us; skew 500 ppm
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:03.682989 25285 init.cc:377] Logged in from keytab as oryx/127.18.44.252@KRBTEST.COM (short username oryx)
I20260923 18:06:03.684397 25285 webserver.cc:460] Webserver started at http://127.18.44.252:42391/ using document root <none> and password file <none>
I20260923 18:06:03.685200 25285 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.685298 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.685761 25285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:03.688329 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/instance:
uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "86536e1e4b1c945a5b7e630ccfe0eab7"
server_key_iv: "ec0087b1fa93a473bc635d8b54c7442c"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.688964 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal/instance:
uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "86536e1e4b1c945a5b7e630ccfe0eab7"
server_key_iv: "ec0087b1fa93a473bc635d8b54c7442c"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.693895 25285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:06:03.697538 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.699733 25285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:06:03.699936 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "86536e1e4b1c945a5b7e630ccfe0eab7"
server_key_iv: "ec0087b1fa93a473bc635d8b54c7442c"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.700075 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.721016 25044 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
I20260923 18:06:03.724288 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:03.727641 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:03.727834 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:03.736158 25285 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.252:43589
I20260923 18:06:03.736621 25439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.252:43589 every 8 connection(s)
I20260923 18:06:03.737304 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
I20260923 18:06:03.741722 25440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:03.745411 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25285
I20260923 18:06:03.747751 25440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.252@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:03.764195 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.750636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.18.44.254:44433)
0923 18:06:03.750819 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.750825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.751358 (+ 533us) server_negotiation.cc:407] Connection header received
0923 18:06:03.751446 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.751450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.751511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.751626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.753015 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.753988 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.755565 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.755916 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.756165 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.759693 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.759697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.759699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.759731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.761803 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.762468 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.762473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.762475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.762583 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.762917 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.762920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.762921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.763227 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.763384 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.763881 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.764023 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 18:06:03.765666 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.750346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38074)
0923 18:06:03.751070 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.751131 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.751313 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.751800 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.751807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.752323 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.752556 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.752572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.754209 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.754212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.755354 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.755369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.756550 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.756558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.756687 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.757244 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.757262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.759510 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.761956 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.761972 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.761980 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.762307 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.762696 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.762699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.762701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.762806 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.763382 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.763390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.763564 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.763763 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.764294 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":176,"thread_start_us":179,"threads_started":1}
I20260923 18:06:03.765666 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.747546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.18.44.252:43589)
0923 18:06:03.748299 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.748327 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.748394 (+ 67us) server_negotiation.cc:407] Connection header received
0923 18:06:03.749370 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.749398 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.749887 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.750311 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.751778 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.753292 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.754649 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.755473 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.755631 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.759025 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.759032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.759046 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.759075 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.760998 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.761792 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.761798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.761804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.761940 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.762378 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.762382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.762384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.763124 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.763430 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.763737 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.764226 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":525,"thread_start_us":227,"threads_started":1}
I20260923 18:06:03.768110 25440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.252@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
I20260923 18:06:03.781877 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.769090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.18.44.253:44899)
0923 18:06:03.769485 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.769489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.769543 (+ 54us) server_negotiation.cc:407] Connection header received
0923 18:06:03.769752 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.769755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.769876 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.769993 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.770617 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.771751 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.773129 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.773382 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.773500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.776820 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.776824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.776827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.776864 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.779242 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.780079 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.780083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.780084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.780236 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.780688 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.780691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.780693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.780986 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.781080 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.781530 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.781664 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":166,"threads_started":1}
I20260923 18:06:03.781925 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.768893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46812)
0923 18:06:03.769452 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.769469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.769588 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.770061 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.770064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.770334 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.770461 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.770468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.772054 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.772059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.772908 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.772932 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.773606 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.773610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.773703 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.774592 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.774607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.776417 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.779564 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.779569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.779571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.779911 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.780428 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.780432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.780435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.780558 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.781151 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.781155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.781309 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.781456 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.781738 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446}
I20260923 18:06:03.783203 25440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.252@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
I20260923 18:06:03.794659 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.783910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.0.0.1:43552)
0923 18:06:03.784096 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.784115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.784232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.784508 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.784510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.784712 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.784832 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.784838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.785730 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.785733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.786398 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.786409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.787386 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.787396 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.787488 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.788281 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.788296 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.790236 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.792296 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.792299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.792301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.792579 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.793130 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.793133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.793134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.793233 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.793833 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.793836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.794160 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.794272 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.794471 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 18:06:03.794711 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.784001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.18.44.252:43589)
0923 18:06:03.784132 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.784135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.784175 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:06:03.784323 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.784325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.784372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.784468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.785006 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.785561 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.786570 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.786986 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.787109 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.790506 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.790510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.790513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.790548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.792110 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.792795 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.792799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.792800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.792899 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.793366 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.793368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.793370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.793585 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.793723 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.794382 (+ 659us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.794547 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:06:03.798949 25440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Bootstrap starting.
I20260923 18:06:03.801522 25440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:03.802258 25440 log.cc:824] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:03.804575 25440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: No bootstrap required, opened a new log
I20260923 18:06:03.807870 25440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } }
I20260923 18:06:03.808117 25440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:03.808151 25440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3e96220d474fad8ffc7ede7d039cb2, State: Initialized, Role: FOLLOWER
I20260923 18:06:03.808588 25440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } }
I20260923 18:06:03.809281 25446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } } }
I20260923 18:06:03.809520 25446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:03.809943 25440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:06:03.817327 25457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:06:03.817503 25457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:06:03.821027 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24781
I20260923 18:06:03.822192 25285 master_runner.cc:428] Detected that this master 0d3e96220d474fad8ffc7ede7d039cb2 is joining an existing cluster
I20260923 18:06:03.822283 25285 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.44.252:43589
I20260923 18:06:03.830760 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:46483
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:44433
--webserver_port=46715
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:06:03.837752 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.825452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46822)
0923 18:06:03.826457 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.826473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.826581 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.827824 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.827828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.828053 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.828186 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.828193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.829317 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.829320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.830677 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.830685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.832095 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.832099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.832194 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.832601 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.832614 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.833701 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.835917 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.835920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.835923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.836182 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.836500 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.836503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.836505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.836616 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.837116 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.837119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.837213 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.837310 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.837489 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"spinlock_wait_cycles":1408,"thread_start_us":214,"threads_started":1}
I20260923 18:06:03.838726 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.827395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.18.44.253:44899)
0923 18:06:03.827532 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.827537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.827553 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:03.827599 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.827603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.827651 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.827754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.828349 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.829170 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.830811 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.831152 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.831284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.833909 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.833914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.833917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.833951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.835782 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.836298 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.836301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.836304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.836393 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.836721 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.836725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.836726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.836979 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.837073 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.838354 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.838544 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:06:03.840796 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.826668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.0.0.1:43562)
0923 18:06:03.832604 (+ 5936us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.832626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.832797 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.833383 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.833387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.833719 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.833852 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.833860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.834867 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.834870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.835770 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.835778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.836448 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.836453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.836544 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.837014 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.837028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.837458 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.839318 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.839320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.839322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.839548 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.839816 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.839818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.839819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.839895 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.840305 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.840307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.840378 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.840461 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.840616 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5831,"thread_start_us":81,"threads_started":1}
I20260923 18:06:03.841251 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.826483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.18.44.252:43589)
0923 18:06:03.826577 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.826581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.832899 (+ 6318us) server_negotiation.cc:407] Connection header received
0923 18:06:03.832964 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.832968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.833020 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.833128 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.833974 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.834711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.835896 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.836204 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.836273 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.837575 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.837579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.837582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.837602 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.839182 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.839640 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.839643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.839645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.839721 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.839983 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.839986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.839988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.840186 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.840264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.841010 (+ 746us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.841126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:06:03.845746 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.833525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.18.44.253:44899)
0923 18:06:03.839334 (+ 5809us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.839338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.839352 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:06:03.839411 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.839414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.839504 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.841595 (+ 2091us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:03.842022 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.843127 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.844494 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.844988 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.845098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.845311 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.845465 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.845561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5712,"thread_start_us":110,"threads_started":1}
I20260923 18:06:03.845746 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.833144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46824)
0923 18:06:03.833266 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.833283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.833373 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.841668 (+ 8295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.841672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.841711 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:03.841861 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.841872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.843311 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.843316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.844317 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.844336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.845151 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.845156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.845264 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.845343 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.845430 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.845580 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 18:06:03.846755 25332 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 18:06:03.847337 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0891315db754d67be66589360f29de3. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } }
I20260923 18:06:03.847506 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:06:03.865506 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.826076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38084)
0923 18:06:03.826925 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.826936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.827024 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.827286 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.827289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.827748 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.828016 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.828025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.829318 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.829320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.830401 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.830411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.831213 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.831217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.831337 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.832027 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.832042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.860080 (+ 28038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.863371 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.863379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.863381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.863750 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.864204 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.864207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.864209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.864314 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.864863 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.864866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.865001 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.865108 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.865287 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260923 18:06:03.865979 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.825850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.18.44.254:44433)
0923 18:06:03.825992 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.825997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.827020 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 18:06:03.827108 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.827111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.827165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.827233 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.828235 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.829177 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.830549 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.830882 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.830974 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.860476 (+ 29502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.860481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.860484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.860535 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.863002 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.863924 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.863929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.863932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.864086 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.864431 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.864434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.864435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.864739 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.864867 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.865206 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.865385 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186761, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.254@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
I20260923 18:06:03.885165 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.872873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.0.0.1:43570)
0923 18:06:03.873286 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.873305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.873426 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.874198 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.874201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.874495 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.874662 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.874670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.875919 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.875923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.876789 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.876804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.877455 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.877459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.877568 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.878369 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.878387 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.880416 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.883063 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.883074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.883077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.883481 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.883888 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.883891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.883893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.883997 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.884490 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.884493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.884631 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.884760 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.884948 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260923 18:06:03.885191 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.873628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.18.44.252:43589)
0923 18:06:03.873769 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.873774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.873789 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:06:03.873897 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.873900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.873965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.874786 (+ 821us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.874908 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.875769 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.876964 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.877305 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.878038 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.880683 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.880689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.880692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.880735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.882847 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.883637 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.883641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.883644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.883772 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.884101 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.884104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.884106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.884382 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.884490 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.884828 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.885003 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:06:03.887101 24719 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.44.252:43589: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
member_type: NON_VOTER
last_known_addr {
host: "127.18.44.252"
port: 43589
}
attrs {
promote: true
}
}
dest_uuid: "d0891315db754d67be66589360f29de3"
cas_config_opid_index: 5
I20260923 18:06:03.887951 24719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } }
I20260923 18:06:03.889310 25280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.890442 25414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:03.892079 25414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 18:06:03.892496 25280 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.44.252:43589
I20260923 18:06:03.892696 25332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 18:06:03.892817 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.892968 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:03.893420 25282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 18:06:03.894029 25282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:03.895188 25285 master.cc:537] Master@127.18.44.252:43589 shutting down...
I20260923 18:06:03.896155 25332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
W20260923 18:06:03.897212 24706 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 -> Peer 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252:43589): Couldn't send request to peer 0d3e96220d474fad8ffc7ede7d039cb2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:06:03.898907 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: NON_VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: true } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.899097 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:06:03.900130 25282 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: attempting to promote NON_VOTER 88baa141f23e40caa57798777380a6b5 to VOTER
I20260923 18:06:03.900719 25282 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } }
I20260923 18:06:03.906037 25332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 LEARNER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 18:06:03.906731 25280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260923 18:06:03.908944 24706 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 -> Peer 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252:43589): Couldn't send request to peer 0d3e96220d474fad8ffc7ede7d039cb2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:06:03.908993 25283 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88baa141f23e40caa57798777380a6b5 (127.18.44.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.909783 25285 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:03.910282 25285 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:03.910524 25285 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: stopping tablet replica
I20260923 18:06:03.910866 25283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.911005 25283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:03.912370 25331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88baa141f23e40caa57798777380a6b5 (127.18.44.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.914426 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.914568 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:03.915015 25281 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:03.915174 25281 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:03.918229 25285 master.cc:552] Master@127.18.44.252:43589 shutdown complete.
I20260923 18:06:03.918319 25285 master_runner.cc:305] Clearing existing system tablet
I20260923 18:06:03.918612 25285 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.918679 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.920477 25285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.921612 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.922056 25285 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.922448 25285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.922544 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "86536e1e4b1c945a5b7e630ccfe0eab7"
server_key_iv: "ec0087b1fa93a473bc635d8b54c7442c"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.925699 25285 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:03.929409 25285 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 18:06:03.929772 25285 log.cc:1196] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 18:06:03.930568 25285 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Deleting consensus metadata
I20260923 18:06:03.930836 25285 master_runner.cc:315] Copying system tablet from 127.18.44.254:44433
I20260923 18:06:03.933132 25285 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: tablet copy: Beginning tablet copy session from remote peer at address 127.18.44.254:44433
I20260923 18:06:03.944948 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.934249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38096)
0923 18:06:03.934709 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:03.934727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:03.934885 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:03.935318 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:03.935321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:03.935585 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:03.935759 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.935768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.936956 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.936960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.937917 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:03.937930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.938698 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.938702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:03.938953 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.939719 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:03.939734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:03.940191 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:03.942735 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.942740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.942743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.943103 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.943621 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:03.943624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:03.943626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:03.943737 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:03.944286 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:03.944289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:03.944430 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 18:06:03.944541 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 18:06:03.944717 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":141,"threads_started":1}
I20260923 18:06:03.945005 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:03.934404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.18.44.254:44433)
0923 18:06:03.934585 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:03.934590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:03.934833 (+ 243us) server_negotiation.cc:407] Connection header received
0923 18:06:03.935018 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:03.935022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:03.935092 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:03.935234 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:03.935923 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.936774 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.938091 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:03.938479 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:03.938675 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:03.940579 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:03.940585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:03.940589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:03.940635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:03.942539 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.943361 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.943365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.943367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.943504 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:03.943862 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:03.943866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:03.943867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:03.944154 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:03.944262 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:03.944644 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 18:06:03.944811 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:06:03.946018 24739 tablet_copy_service.cc:140] P d0891315db754d67be66589360f29de3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d3e96220d474fad8ffc7ede7d039cb2 ({username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:38096)
I20260923 18:06:03.946130 24739 tablet_copy_service.cc:161] P d0891315db754d67be66589360f29de3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d3e96220d474fad8ffc7ede7d039cb2 at {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:38096: session id = 0d3e96220d474fad8ffc7ede7d039cb2-00000000000000000000000000000000
I20260923 18:06:03.948017 24739 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 18:06:03.949895 25285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 18:06:03.952911 25285 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: tablet copy: Starting download of 0 data blocks...
I20260923 18:06:03.953284 25285 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: tablet copy: Starting download of 1 WAL segments...
I20260923 18:06:03.956194 25285 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 18:06:03.959146 24739 tablet_copy_service.cc:342] P d0891315db754d67be66589360f29de3: Request end of tablet copy session 0d3e96220d474fad8ffc7ede7d039cb2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:38096
I20260923 18:06:03.959331 24739 tablet_copy_service.cc:433] P d0891315db754d67be66589360f29de3: ending tablet copy session 0d3e96220d474fad8ffc7ede7d039cb2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d3e96220d474fad8ffc7ede7d039cb2
I20260923 18:06:03.968802 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:03.971313 25477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:03.971657 25478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.971931 25285 server_base.cc:1034] running on GCE node
W20260923 18:06:03.972272 25480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:03.972596 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:03.972982 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:03.974166 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790186763974139 us; error 62 us; skew 500 ppm
Sep 23 18:06:03 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186763, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:03.976948 25285 init.cc:377] Logged in from keytab as oryx/127.18.44.252@KRBTEST.COM (short username oryx)
I20260923 18:06:03.977607 25285 webserver.cc:460] Webserver started at http://127.18.44.252:42303/ using document root <none> and password file <none>
I20260923 18:06:03.977931 25285 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:03.978011 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:03.980070 25285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 18:06:03.981330 25487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:03.981743 25285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:03.981854 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
format_stamp: "Formatted at 2026-09-23 18:06:03 on dist-test-slave-rvjj"
server_key: "86536e1e4b1c945a5b7e630ccfe0eab7"
server_key_iv: "ec0087b1fa93a473bc635d8b54c7442c"
server_key_version: "encryptionkey@0"
I20260923 18:06:03.981954 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:03.995699 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:03.998605 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:04.003686 25285 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.252:43589
I20260923 18:06:04.004545 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
I20260923 18:06:04.004840 25539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.252:43589 every 8 connection(s)
I20260923 18:06:04.006150 25540 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:06:04.007053 25540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Bootstrap starting.
I20260923 18:06:04.017639 25540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:06:04.017979 25540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Bootstrap complete.
I20260923 18:06:04.018555 25540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } }
I20260923 18:06:04.018684 25540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d3e96220d474fad8ffc7ede7d039cb2, State: Initialized, Role: LEARNER
I20260923 18:06:04.018790 25540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } }
I20260923 18:06:04.019117 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:04.019232 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:06:04.019306 25540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:04.022639 25557 catalog_manager.cc:1259] Loaded cluster ID: ba34ba7ebde84ca49875e1e917af0539
I20260923 18:06:04.022729 25557 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: loading cluster ID for follower catalog manager: success
I20260923 18:06:04.024140 25178 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
I20260923 18:06:04.025085 25557 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: acquiring CA information for follower catalog manager: success
I20260923 18:06:04.026571 25557 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 18:06:04.027472 25466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:04.028072 25466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:04.028173 25466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:04.033910 25466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:04.034058 25466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:04.034137 25466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:04.042791 25466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:46483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=46715
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:04.044211 25466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:04.045425 25466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:06:04.057181 25466 server_base.cc:1034] running on GCE node
W20260923 18:06:04.057286 25559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:04.057225 25560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:04.057181 25562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:04.057955 25466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:04.058784 25466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:04.060000 25466 hybrid_clock.cc:648] HybridClock initialized: now 1790186764059977 us; error 43 us; skew 500 ppm
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:04.063153 25466 init.cc:377] Logged in from keytab as oryx/127.18.44.193@KRBTEST.COM (short username oryx)
I20260923 18:06:04.064266 25466 webserver.cc:460] Webserver started at http://127.18.44.193:46715/ using document root <none> and password file <none>
I20260923 18:06:04.064888 25466 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:04.064976 25466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:04.069461 25466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:06:04.072028 25569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.073673 25466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:06:04.073817 25466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "c3cd43e195e24ea2af9092dcf35ed533"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "febb48fe28ef24cc3a944707a5267c50"
server_key_iv: "785cb29cb1300b4fd91eaebd75ca0989"
server_key_version: "encryptionkey@0"
I20260923 18:06:04.074523 25466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:04.088882 25466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:04.092206 25466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:04.092396 25466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:04.093032 25466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:04.094018 25466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:04.094072 25466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.094116 25466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:04.094134 25466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.103839 25466 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:46483
I20260923 18:06:04.103853 25682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:46483 every 8 connection(s)
I20260923 18:06:04.104975 25466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:04.112538 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25466
I20260923 18:06:04.112753 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24915
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:04.123232 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:45397
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:44433
--webserver_port=35337
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:06:04.123750 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.106944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38539 (local address 127.18.44.253:44899)
0923 18:06:04.107081 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.107085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.108294 (+ 1209us) server_negotiation.cc:407] Connection header received
0923 18:06:04.109607 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.109611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.109673 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.109759 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.111393 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.112239 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114187 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114567 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114652 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.119214 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.119218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.119220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.119275 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.120986 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121709 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.121713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.121715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.121806 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.122132 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.122135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.122137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.122636 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.122725 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.123434 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.123566 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:06:04.123761 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.108820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:37543 (local address 127.18.44.254:44433)
0923 18:06:04.108948 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.108952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.109019 (+ 67us) server_negotiation.cc:407] Connection header received
0923 18:06:04.110083 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.110087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.110140 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.110223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.112405 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.113201 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114305 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114644 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114718 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.119004 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.119008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.119011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.119044 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.121054 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121707 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.121711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.121712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.121806 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.122130 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.122134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.122135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.122368 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.122429 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.123413 (+ 984us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.123566 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:06:04.124500 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.106970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:41687 (local address 127.18.44.252:43589)
0923 18:06:04.107311 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.107316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.108259 (+ 943us) server_negotiation.cc:407] Connection header received
0923 18:06:04.109815 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.109818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.109870 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.109949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.111453 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.112478 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114242 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114595 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114695 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.118563 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.118567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.118570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.118605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.120760 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121527 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.121532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.121533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.121630 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.122168 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.122171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.122173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.122430 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.122530 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.123708 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.124337 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260923 18:06:04.125727 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.107559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.18.44.193:38539)
0923 18:06:04.108087 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.108143 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.109394 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.109966 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.109975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.110556 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.111217 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.111232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.112408 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.112412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.114020 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.114032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114902 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.115437 (+ 531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.116391 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.116404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.119037 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.121124 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.121129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.121590 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.121913 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.121918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.122038 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.122743 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.122746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.123125 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.123965 (+ 840us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.124351 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":568,"spinlock_wait_cycles":22656,"thread_start_us":131,"threads_started":1}
I20260923 18:06:04.125936 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.107559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.18.44.193:41687)
0923 18:06:04.108114 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.108144 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.109403 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.110000 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.110004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.110560 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.111212 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.111312 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.112600 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.112604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.114018 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.114031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114902 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.115272 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.116102 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.116125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.118239 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.120930 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.120937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.120961 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.121407 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.121720 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.121725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.122039 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.122587 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.122595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.122997 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.123231 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.125199 (+ 1968us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":121,"spinlock_wait_cycles":126976,"thread_start_us":139,"threads_started":1}
I20260923 18:06:04.125727 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.108698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:37543)
0923 18:06:04.108949 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.108961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.109403 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.110263 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.110266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.110556 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.111217 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.111232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.113355 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.113358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.114180 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.114187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.114902 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.114905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.115371 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.116111 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.116132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.118816 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.121194 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121196 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.121198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.121590 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.121913 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.121916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.121918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.122026 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.122791 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.122793 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.123125 (+ 332us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.123909 (+ 784us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.124419 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":780,"spinlock_wait_cycles":30080,"thread_start_us":84,"threads_started":1}
I20260923 18:06:04.127758 25685 heartbeater.cc:347] Connected to a master server at 127.18.44.253:44899
I20260923 18:06:04.127843 25684 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:04.128036 25685 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.128034 25684 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.128588 25683 heartbeater.cc:347] Connected to a master server at 127.18.44.252:43589
I20260923 18:06:04.128692 25685 heartbeater.cc:511] Master 127.18.44.253:44899 requested a full tablet report, sending...
I20260923 18:06:04.128724 25683 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.128712 25684 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
I20260923 18:06:04.129065 25683 heartbeater.cc:511] Master 127.18.44.252:43589 requested a full tablet report, sending...
I20260923 18:06:04.130362 24719 ts_manager.cc:194] Re-registered known tserver with Master: c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193:46483)
I20260923 18:06:04.130594 25504 ts_manager.cc:194] Registered new tserver with Master: c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193:46483)
I20260923 18:06:04.130594 25322 ts_manager.cc:194] Registered new tserver with Master: c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193:46483)
I20260923 18:06:04.131224 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:37543
W20260923 18:06:04.261437 25691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:04.261847 25691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:04.261917 25691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:04.266494 25691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:04.266583 25691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:04.266646 25691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:04.273428 25691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:45397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=35337
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:04.274616 25691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:04.275866 25691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:04.286963 25697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:04.286966 25699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:04.286966 25696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:04.287456 25691 server_base.cc:1034] running on GCE node
I20260923 18:06:04.287914 25691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:04.288620 25691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:04.289799 25691 hybrid_clock.cc:648] HybridClock initialized: now 1790186764289788 us; error 29 us; skew 500 ppm
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:04.292856 25691 init.cc:377] Logged in from keytab as oryx/127.18.44.194@KRBTEST.COM (short username oryx)
I20260923 18:06:04.293994 25691 webserver.cc:460] Webserver started at http://127.18.44.194:35337/ using document root <none> and password file <none>
I20260923 18:06:04.294620 25691 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:04.294718 25691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:04.299499 25691 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:06:04.301811 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.297722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.0.0.1:43572)
0923 18:06:04.297856 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.297873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.297971 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.298230 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.298233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.298250 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.298358 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.298365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.299549 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.299552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.300452 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.300464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.301374 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.301377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.301455 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.301478 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.301558 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.301697 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 18:06:04.301815 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.297864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.18.44.252:43589)
0923 18:06:04.297985 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.297990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.298006 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:04.298060 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.298062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.298113 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.298204 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.298513 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.299378 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.300604 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.301231 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.301347 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.301505 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.301578 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.301667 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:06:04.302201 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.302629 25514 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 18:06:04.303037 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0891315db754d67be66589360f29de3. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:04.303196 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:06:04.303985 25691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:06:04.304121 25691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "bc9aab5e9576496e809caf334432579b"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "403327d65686d4835787e05668e1168c"
server_key_iv: "00be751ce9a6c83eebfde63681e52161"
server_key_version: "encryptionkey@0"
I20260923 18:06:04.304761 25691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:04.324365 25691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:04.328210 25691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:04.328403 25691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:04.329108 25691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:04.330339 25691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:04.330399 25691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.330448 25691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:04.330476 25691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.340000 25691 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:45397
I20260923 18:06:04.340098 25819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:45397 every 8 connection(s)
I20260923 18:06:04.341115 25691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:04.343724 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25691
I20260923 18:06:04.343894 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25049
I20260923 18:06:04.350999 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:46033
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:44433
--webserver_port=37847
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
I20260923 18:06:04.360087 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.343206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:57441 (local address 127.18.44.252:43589)
0923 18:06:04.343395 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.343400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.345325 (+ 1925us) server_negotiation.cc:407] Connection header received
0923 18:06:04.346784 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.346788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.346842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.346895 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.349042 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.349872 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.351919 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352273 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.355994 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.355998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.356000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.356027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.357547 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.358232 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.358235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.358237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.358312 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.358675 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.358678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.358680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.358977 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.359070 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.359776 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.359922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 18:06:04.361102 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.343719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.18.44.194:57441)
0923 18:06:04.345101 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.345184 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.346524 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.347358 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.347369 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.348030 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.348847 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.348869 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.350015 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.350018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.351748 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.351763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352591 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.352867 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.353671 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.353699 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.355749 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.357748 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.357754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.357770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.358116 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.358411 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.358414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.358416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.358575 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.359071 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.359078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.359424 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.359694 (+ 270us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.360151 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":369,"spinlock_wait_cycles":33280,"thread_start_us":191,"threads_started":1}
I20260923 18:06:04.362322 25820 heartbeater.cc:347] Connected to a master server at 127.18.44.252:43589
I20260923 18:06:04.362566 25820 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.363109 25820 heartbeater.cc:511] Master 127.18.44.252:43589 requested a full tablet report, sending...
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:04.364430 25504 ts_manager.cc:194] Registered new tserver with Master: bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
I20260923 18:06:04.369349 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.343223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:58839 (local address 127.18.44.253:44899)
0923 18:06:04.343400 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.343404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.345483 (+ 2079us) server_negotiation.cc:407] Connection header received
0923 18:06:04.346788 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.346793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.346847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.346896 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.349037 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.349770 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352280 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352609 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352683 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.366126 (+ 13443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.366130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.366133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.366166 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.367601 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.368056 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.368060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.368061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.368137 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.368437 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.368440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.368442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.368673 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.368748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.369079 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.369194 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 18:06:04.369365 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.344936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:45161)
0923 18:06:04.346559 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.346573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.346724 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.347358 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.347369 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.348030 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.348852 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.348869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.350015 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.350018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.351748 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.351763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352591 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.352898 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.353742 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.353755 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.365421 (+ 11666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.367566 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.367579 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.367582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.367878 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.368190 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.368193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.368195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.368303 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.368744 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.368748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.368862 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.368958 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.369131 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1558,"mutex_wait_us":314,"spinlock_wait_cycles":9600,"thread_start_us":88,"threads_started":1}
I20260923 18:06:04.369371 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.345781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:45161 (local address 127.18.44.254:44433)
0923 18:06:04.345912 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.345917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.346653 (+ 736us) server_negotiation.cc:407] Connection header received
0923 18:06:04.346819 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.346822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.346867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.346914 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.349041 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.349872 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.351919 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352290 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352355 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.365626 (+ 13271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.365631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.365635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.365674 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.367445 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.367997 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.368001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.368003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.368100 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.368399 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.368402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.368404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.368646 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.368745 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.369048 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.369177 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:06:04.369925 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.343719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.18.44.194:58839)
0923 18:06:04.345390 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.345403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.346534 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.347359 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.347369 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.348042 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.348847 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.348869 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.349963 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.349966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.352113 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.352125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.352833 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.352837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.352973 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.353679 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.353698 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.365960 (+ 12262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.367697 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.367701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.367703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.367947 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.368230 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.368233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.368235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.368317 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.368812 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.368815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.368892 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.368967 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.369710 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"mutex_wait_us":776,"spinlock_wait_cycles":77056,"thread_start_us":272,"threads_started":1}
I20260923 18:06:04.370497 25821 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:04.370610 25821 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.370821 25821 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
I20260923 18:06:04.371210 25822 heartbeater.cc:347] Connected to a master server at 127.18.44.253:44899
I20260923 18:06:04.371315 25822 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.371452 25822 heartbeater.cc:511] Master 127.18.44.253:44899 requested a full tablet report, sending...
I20260923 18:06:04.371533 24719 ts_manager.cc:194] Re-registered known tserver with Master: bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:04.372076 25322 ts_manager.cc:194] Registered new tserver with Master: bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:04.372288 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:45161
W20260923 18:06:04.474776 25827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:04.475126 25827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:04.475188 25827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:04.479930 25827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:04.480036 25827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:04.480134 25827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:04.486769 25827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:46033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=37847
--tserver_master_addrs=127.18.44.254:44433,127.18.44.253:44899,127.18.44.252:43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:04.488153 25827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:04.489246 25827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:04.500612 25835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:04.500598 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:04.500656 25827 server_base.cc:1034] running on GCE node
W20260923 18:06:04.501114 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:04.501571 25827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:04.502321 25827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:04.503477 25827 hybrid_clock.cc:648] HybridClock initialized: now 1790186764503467 us; error 32 us; skew 500 ppm
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:04.512908 25827 init.cc:377] Logged in from keytab as oryx/127.18.44.195@KRBTEST.COM (short username oryx)
I20260923 18:06:04.514004 25827 webserver.cc:460] Webserver started at http://127.18.44.195:37847/ using document root <none> and password file <none>
I20260923 18:06:04.514623 25827 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:04.514720 25827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:04.519356 25827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:06:04.521862 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.523725 25827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:06:04.523870 25827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "7133a7fcb4fe45a08470cc277e8bca73"
format_stamp: "Formatted at 2026-09-23 18:06:02 on dist-test-slave-rvjj"
server_key: "01ed7fb37c83dbd011c504b074b1707c"
server_key_iv: "940d651cff646177363ef335b81142eb"
server_key_version: "encryptionkey@0"
I20260923 18:06:04.524484 25827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:04.541687 25827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:04.545749 25827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:04.545956 25827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:04.546633 25827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:04.547814 25827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:04.547885 25827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.547940 25827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:04.547966 25827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:04.559700 25827 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46033
I20260923 18:06:04.559792 25955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46033 every 8 connection(s)
I20260923 18:06:04.560801 25827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:04.569778 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 25827
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.253@KRBTEST.COM
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
Sep 23 18:06:04 dist-test-slave-rvjj krb5kdc[24674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186764, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.252@KRBTEST.COM
I20260923 18:06:04.579035 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.562715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:41859 (local address 127.18.44.253:44899)
0923 18:06:04.562856 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.562861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.564214 (+ 1353us) server_negotiation.cc:407] Connection header received
0923 18:06:04.565534 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.565537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.565592 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.565661 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.567187 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.568051 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.569716 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570066 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570140 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.574775 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.574779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.574782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.574823 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.576404 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577386 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577472 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577759 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577970 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.578029 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.578710 (+ 681us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.578860 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:06:04.579181 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.562733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:41475 (local address 127.18.44.254:44433)
0923 18:06:04.562892 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.562898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.564214 (+ 1316us) server_negotiation.cc:407] Connection header received
0923 18:06:04.565426 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.565430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.565495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.565567 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.567222 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.568073 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.569718 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570067 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570140 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.574360 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.574364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.574367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.574392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.576747 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577170 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577261 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577619 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577838 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.577929 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.578917 (+ 988us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.579033 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 18:06:04.580420 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.563215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:41475)
0923 18:06:04.564031 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.564091 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.565222 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.565786 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.565848 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.566383 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.567036 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.567046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.568215 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.568218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.569606 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.569613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570371 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.570639 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.571520 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.571547 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.574142 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.576858 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.576861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.576863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.577091 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.577379 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.577384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.577517 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.577985 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.577992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.578411 (+ 419us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.578637 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.579943 (+ 1306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":1434,"spinlock_wait_cycles":112256,"thread_start_us":141,"threads_started":1}
I20260923 18:06:04.580435 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.563215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.18.44.195:41859)
0923 18:06:04.564031 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.564091 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.565157 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.565787 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.565797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.566382 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.567029 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.567046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.568147 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.568150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.569575 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.569587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570466 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.570630 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.571519 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.571545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.574614 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.576504 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.576507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.576509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.577270 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.577581 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.577585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.577661 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.578097 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.578099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.578411 (+ 312us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.578637 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.579037 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":233,"spinlock_wait_cycles":39936,"thread_start_us":134,"threads_started":1}
I20260923 18:06:04.581419 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.564489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:40841 (local address 127.18.44.252:43589)
0923 18:06:04.564604 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.564609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.564887 (+ 278us) server_negotiation.cc:407] Connection header received
0923 18:06:04.565409 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.565412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.565470 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.565523 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.567336 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.567956 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.569715 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570024 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570125 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.574688 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.574691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.574694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.574719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.576323 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577087 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577177 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577588 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.577591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.577593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.577833 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.577903 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.581118 (+ 3215us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.581271 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:06:04.581985 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.564357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.18.44.195:40841)
0923 18:06:04.564754 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.564779 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.565170 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.565787 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.565797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.566383 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:04.567040 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.567046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.568106 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.568110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.569559 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.569573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.570371 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.570375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.570667 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.571669 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:04.571685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:04.574520 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:04.576476 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.576484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.576505 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.576974 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.577282 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:04.577285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:04.577287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:04.577487 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:04.577984 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:04.577992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:04.580885 (+ 2893us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.581595 (+ 710us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.581816 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":2892,"spinlock_wait_cycles":19456,"thread_start_us":99,"threads_started":1}
I20260923 18:06:04.582299 25958 heartbeater.cc:347] Connected to a master server at 127.18.44.253:44899
I20260923 18:06:04.582379 25957 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44433
I20260923 18:06:04.582576 25958 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.582576 25957 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.583033 25956 heartbeater.cc:347] Connected to a master server at 127.18.44.252:43589
I20260923 18:06:04.583114 25956 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:04.583321 25958 heartbeater.cc:511] Master 127.18.44.253:44899 requested a full tablet report, sending...
I20260923 18:06:04.583320 25957 heartbeater.cc:511] Master 127.18.44.254:44433 requested a full tablet report, sending...
I20260923 18:06:04.583320 25956 heartbeater.cc:511] Master 127.18.44.252:43589 requested a full tablet report, sending...
I20260923 18:06:04.584843 25504 ts_manager.cc:194] Registered new tserver with Master: 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033)
I20260923 18:06:04.585004 24719 ts_manager.cc:194] Re-registered known tserver with Master: 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033)
I20260923 18:06:04.585793 24719 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:41475
I20260923 18:06:04.586630 25322 ts_manager.cc:194] Registered new tserver with Master: 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033)
I20260923 18:06:04.589824 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.572532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.18.44.252:43589)
0923 18:06:04.572792 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.572797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.573367 (+ 570us) server_negotiation.cc:407] Connection header received
0923 18:06:04.573477 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.573480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.573549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.573634 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.575542 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.576395 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.577541 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.577886 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.577972 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.583971 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.583975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.583978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.584010 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.587902 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.588401 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.588405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.588408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.588506 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.588856 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.588861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.588863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.589061 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.589209 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.589467 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.589608 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260923 18:06:04.590629 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.573358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.18.44.253:44899)
0923 18:06:04.579238 (+ 5880us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.579242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.579388 (+ 146us) server_negotiation.cc:407] Connection header received
0923 18:06:04.579426 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.579429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.579474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.579572 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.581506 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.582370 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.584643 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.584957 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.585066 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.586912 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.586917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.586920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.586943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.588555 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.589036 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.589040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.589042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.589132 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.589445 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.589448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.589450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.589664 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.589781 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.590371 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.590475 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5815,"thread_start_us":103,"threads_started":1}
I20260923 18:06:04.592684 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.572105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.18.44.254:44433)
0923 18:06:04.573608 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.573612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.573627 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:06:04.573670 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.573673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.573720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.573808 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.577026 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.579018 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.580485 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.580846 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.580956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.587600 (+ 6644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.587604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.587607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.587636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.589348 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.589848 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.589851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.589853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.589976 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.590307 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.590311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.590312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.590555 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.590720 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.592400 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.592534 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":67,"threads_started":1}
I20260923 18:06:04.596108 24719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:04.599582 24719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:04.602442 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: NON_VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: true } } }
I20260923 18:06:04.602608 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:04.606189 25280 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: attempting to promote NON_VOTER 0d3e96220d474fad8ffc7ede7d039cb2 to VOTER
I20260923 18:06:04.606577 25280 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:04.607813 25331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 18:06:04.608198 25513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 LEARNER]: Refusing update from remote peer d0891315db754d67be66589360f29de3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 18:06:04.608610 25974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 18:06:04.608856 25975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 18:06:04.610833 25280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.611500 25331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.611528 25513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.612372 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.612501 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:04.612658 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.612728 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:04.612830 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.612900 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:04.613481 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.613605 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:04.614936 25983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0891315db754d67be66589360f29de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:04.615181 25983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:04.621115 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.616242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46033 (local address 127.0.0.1:57072)
0923 18:06:04.616435 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.616452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.616538 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.617321 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.617325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.617343 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.617465 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.617473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.618773 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.618775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.619642 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.619652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.620702 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.620706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.620797 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.620819 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.620888 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.620972 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2}
I20260923 18:06:04.621158 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.616377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.18.44.195:46033)
0923 18:06:04.616820 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.616836 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.616860 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:06:04.616945 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.616951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.617167 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.617344 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.617571 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.618656 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.619762 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.620552 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.620656 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.620815 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.620925 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.621015 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":120,"threads_started":1}
I20260923 18:06:04.622681 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.616393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:46483 (local address 127.0.0.1:41084)
0923 18:06:04.616882 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.616897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.616990 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.619330 (+ 2340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.619333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.619352 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.619458 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.619465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.620581 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.620584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.621500 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.621511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.622262 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.622266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.622357 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.622380 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.622455 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.622534 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":113,"threads_started":1}
I20260923 18:06:04.622699 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.616816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.18.44.193:46483)
0923 18:06:04.618944 (+ 2128us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.618958 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.618976 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:06:04.619033 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.619037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.619159 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.619342 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.619577 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.620483 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.621602 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.622136 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.622231 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.622380 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.622485 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.622577 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":74,"threads_started":1}
I20260923 18:06:04.624223 25890 tablet_service.cc:1467] Processing CreateTablet for tablet 76529832705b4ef797524875267c1938 (DEFAULT_TABLE table=test-table [id=c65a7d65b1f0432f8a9cd3437555ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:04.625447 25890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76529832705b4ef797524875267c1938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:04.625867 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.617383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:45397 (local address 127.0.0.1:49346)
0923 18:06:04.618019 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.618030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.618120 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.618581 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.618583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.618601 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.618706 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.618712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.623109 (+ 4397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.623112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.624300 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.624312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.625380 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.625384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.625475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.625553 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.625630 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.625719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":92,"threads_started":1}
I20260923 18:06:04.625909 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.617553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.18.44.194:45397)
0923 18:06:04.617948 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.617959 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.618116 (+ 157us) server_negotiation.cc:407] Connection header received
0923 18:06:04.618224 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.618230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.618445 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.618592 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.619574 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.623005 (+ 3431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.624456 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.625240 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.625357 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.625496 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.625664 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.625765 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":88,"threads_started":1}
I20260923 18:06:04.626001 25617 tablet_service.cc:1467] Processing CreateTablet for tablet 76529832705b4ef797524875267c1938 (DEFAULT_TABLE table=test-table [id=c65a7d65b1f0432f8a9cd3437555ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:04.627002 25617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76529832705b4ef797524875267c1938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:04.633127 25990 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Bootstrap starting.
I20260923 18:06:04.634603 25754 tablet_service.cc:1467] Processing CreateTablet for tablet 76529832705b4ef797524875267c1938 (DEFAULT_TABLE table=test-table [id=c65a7d65b1f0432f8a9cd3437555ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:04.635862 25990 tablet_bootstrap.cc:654] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:04.635932 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76529832705b4ef797524875267c1938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:04.636831 25991 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Bootstrap starting.
I20260923 18:06:04.636854 25990 log.cc:824] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:04.639000 25991 tablet_bootstrap.cc:654] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:04.639591 25990 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: No bootstrap required, opened a new log
I20260923 18:06:04.639828 25990 ts_tablet_manager.cc:1397] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:06:04.639848 25991 log.cc:824] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:04.642197 25991 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: No bootstrap required, opened a new log
I20260923 18:06:04.642411 25991 ts_tablet_manager.cc:1397] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:06:04.644606 25990 raft_consensus.cc:348] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.644915 25990 raft_consensus.cc:374] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:04.644968 25990 raft_consensus.cc:729] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133a7fcb4fe45a08470cc277e8bca73, State: Initialized, Role: FOLLOWER
I20260923 18:06:04.645465 25990 consensus_queue.cc:260] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.645643 25991 raft_consensus.cc:348] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.645938 25991 raft_consensus.cc:374] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:04.645869 25994 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Bootstrap starting.
I20260923 18:06:04.646008 25991 raft_consensus.cc:729] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cd43e195e24ea2af9092dcf35ed533, State: Initialized, Role: FOLLOWER
I20260923 18:06:04.646445 25991 consensus_queue.cc:260] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.646632 25990 ts_tablet_manager.cc:1428] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 18:06:04.647447 25991 ts_tablet_manager.cc:1428] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:06:04.647805 25684 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
I20260923 18:06:04.648834 25994 tablet_bootstrap.cc:654] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:04.649399 25957 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
I20260923 18:06:04.650336 25994 log.cc:824] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:04.655674 25994 tablet_bootstrap.cc:492] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: No bootstrap required, opened a new log
I20260923 18:06:04.655934 25994 ts_tablet_manager.cc:1397] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 18:06:04.659750 25994 raft_consensus.cc:348] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.660033 25994 raft_consensus.cc:374] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:04.660077 25994 raft_consensus.cc:729] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9aab5e9576496e809caf334432579b, State: Initialized, Role: FOLLOWER
I20260923 18:06:04.660552 25994 consensus_queue.cc:260] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.661304 25821 heartbeater.cc:503] Master 127.18.44.254:44433 was elected leader, sending a full tablet report...
I20260923 18:06:04.661597 25994 ts_tablet_manager.cc:1428] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:06:04.665575 25996 raft_consensus.cc:482] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:04.665839 25996 raft_consensus.cc:504] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.667023 25996 leader_election.cc:288] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033), bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:04.672711 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.667417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38169 (local address 127.18.44.194:45397)
0923 18:06:04.667541 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.667546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.667862 (+ 316us) server_negotiation.cc:407] Connection header received
0923 18:06:04.667947 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.667950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.667994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.668067 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.668456 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.669322 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.671310 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.672031 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.672172 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.672227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.672524 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.672599 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:06:04.672737 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.667341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:45397 (local address 127.18.44.193:38169)
0923 18:06:04.667771 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.667784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.667867 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.668103 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.668106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.668128 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.668338 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.668344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.669513 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.669516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.671091 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.671104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.672269 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.672274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.672398 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.672423 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.672495 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.672569 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":94,"spinlock_wait_cycles":2176,"thread_start_us":108,"threads_started":1}
I20260923 18:06:04.673363 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.667237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46033 (local address 127.18.44.193:57127)
0923 18:06:04.667597 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:04.667614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:04.667727 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:04.668064 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:04.668068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:04.668092 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:04.668305 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.668313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.669513 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.669517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.671088 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:04.671104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.672922 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.672926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:04.673028 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.673052 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:06:04.673129 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:06:04.673216 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":172,"spinlock_wait_cycles":6400,"thread_start_us":134,"threads_started":1}
I20260923 18:06:04.673370 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.667686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:57127 (local address 127.18.44.195:46033)
0923 18:06:04.667795 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.667798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.667811 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:06:04.667863 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.667866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.667916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.668003 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:04.668457 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.669343 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.671351 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.672025 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.672172 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.672238 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.673171 (+ 933us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.673274 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:06:04.673540 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76529832705b4ef797524875267c1938" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9aab5e9576496e809caf334432579b" is_pre_election: true
I20260923 18:06:04.674000 25774 raft_consensus.cc:2454] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 0.
I20260923 18:06:04.674647 25573 leader_election.cc:302] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9aab5e9576496e809caf334432579b, c3cd43e195e24ea2af9092dcf35ed533; no voters:
I20260923 18:06:04.675135 25996 raft_consensus.cc:2781] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:04.675246 25996 raft_consensus.cc:482] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:04.675331 25996 raft_consensus.cc:3037] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:04.675069 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76529832705b4ef797524875267c1938" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133a7fcb4fe45a08470cc277e8bca73" is_pre_election: true
I20260923 18:06:04.675613 25910 raft_consensus.cc:2454] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 0.
I20260923 18:06:04.676692 25996 raft_consensus.cc:504] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.677076 25996 leader_election.cc:288] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 election: Requested vote from peers 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033), bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:04.677456 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76529832705b4ef797524875267c1938" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133a7fcb4fe45a08470cc277e8bca73"
I20260923 18:06:04.677589 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76529832705b4ef797524875267c1938" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9aab5e9576496e809caf334432579b"
I20260923 18:06:04.677634 25910 raft_consensus.cc:3037] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:04.677733 25774 raft_consensus.cc:3037] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:04.679050 25774 raft_consensus.cc:2454] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 1.
I20260923 18:06:04.679050 25910 raft_consensus.cc:2454] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 1.
I20260923 18:06:04.679540 25570 leader_election.cc:302] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7133a7fcb4fe45a08470cc277e8bca73, bc9aab5e9576496e809caf334432579b, c3cd43e195e24ea2af9092dcf35ed533; no voters:
I20260923 18:06:04.679860 25996 raft_consensus.cc:2781] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:04.680106 25996 raft_consensus.cc:686] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 1 LEADER]: Becoming Leader. State: Replica: c3cd43e195e24ea2af9092dcf35ed533, State: Running, Role: LEADER
I20260923 18:06:04.680470 25996 consensus_queue.cc:237] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:04.684288 24718 catalog_manager.cc:5495] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "c3cd43e195e24ea2af9092dcf35ed533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } health_report { overall_health: UNKNOWN } } }
I20260923 18:06:04.714483 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.710186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.18.44.193:46483)
0923 18:06:04.710354 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.710358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.710371 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:06:04.710450 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.710452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.710508 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.710601 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:04.710944 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.711551 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.712520 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.712782 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.712911 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.713316 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:04.713871 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:04.713981 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.714168 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.714273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:06:04.721447 25774 raft_consensus.cc:1261] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Refusing update from remote peer c3cd43e195e24ea2af9092dcf35ed533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:04.721447 25910 raft_consensus.cc:1261] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Refusing update from remote peer c3cd43e195e24ea2af9092dcf35ed533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:04.722350 25996 consensus_queue.cc:1035] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:04.722590 26001 consensus_queue.cc:1035] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:04.731613 26004 mvcc.cc:205] Tried to move back new op lower bound from 7332604988290564096 to 7332604988134150144. Current Snapshot: MvccSnapshot[applied={T|T < 7332604988290564096}]
I20260923 18:06:04.732892 26006 mvcc.cc:205] Tried to move back new op lower bound from 7332604988290564096 to 7332604988134150144. Current Snapshot: MvccSnapshot[applied={T|T < 7332604988290564096}]
I20260923 18:06:04.746178 24718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:04.746481 24718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:04.751596 24718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0891315db754d67be66589360f29de3: Sending DeleteTablet for 3 replicas of tablet 76529832705b4ef797524875267c1938
I20260923 18:06:04.752451 25890 tablet_service.cc:1514] Processing DeleteTablet for tablet 76529832705b4ef797524875267c1938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:04 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:57072
I20260923 18:06:04.752544 25617 tablet_service.cc:1514] Processing DeleteTablet for tablet 76529832705b4ef797524875267c1938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:04 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:41084
I20260923 18:06:04.753348 26013 tablet_replica.cc:330] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: stopping tablet replica
I20260923 18:06:04.753657 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet 76529832705b4ef797524875267c1938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:04 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:49346
I20260923 18:06:04.753721 26013 raft_consensus.cc:2229] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:04.754205 26013 raft_consensus.cc:2258] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:04.754317 26015 tablet_replica.cc:330] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: stopping tablet replica
I20260923 18:06:04.754627 26014 tablet_replica.cc:330] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: stopping tablet replica
I20260923 18:06:04.754670 26015 raft_consensus.cc:2229] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:04.754938 26014 raft_consensus.cc:2229] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:04.755132 26015 raft_consensus.cc:2258] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:04.755241 26014 raft_consensus.cc:2258] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:04.756860 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 24687
I20260923 18:06:04.757118 26013 ts_tablet_manager.cc:1905] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:04.757143 26015 ts_tablet_manager.cc:1905] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:04.757162 26014 ts_tablet_manager.cc:1905] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:04.760069 26015 ts_tablet_manager.cc:1918] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:04.760069 26013 ts_tablet_manager.cc:1918] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:04.760070 26014 ts_tablet_manager.cc:1918] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:04.760181 26014 log.cc:1196] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/76529832705b4ef797524875267c1938
I20260923 18:06:04.760183 26013 log.cc:1196] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/76529832705b4ef797524875267c1938
I20260923 18:06:04.760345 26015 log.cc:1196] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/76529832705b4ef797524875267c1938
I20260923 18:06:04.760501 26014 ts_tablet_manager.cc:1939] T 76529832705b4ef797524875267c1938 P 7133a7fcb4fe45a08470cc277e8bca73: Deleting consensus metadata
I20260923 18:06:04.760525 26013 ts_tablet_manager.cc:1939] T 76529832705b4ef797524875267c1938 P c3cd43e195e24ea2af9092dcf35ed533: Deleting consensus metadata
I20260923 18:06:04.760756 26015 ts_tablet_manager.cc:1939] T 76529832705b4ef797524875267c1938 P bc9aab5e9576496e809caf334432579b: Deleting consensus metadata
W20260923 18:06:04.766091 25570 connection.cc:535] server connection from 127.0.0.1:41084 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 18:06:04.766114 25710 connection.cc:535] server connection from 127.0.0.1:49346 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 18:06:04.766561 25846 connection.cc:535] server connection from 127.0.0.1:57072 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:06:04.770129 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.755386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.18.44.253:44899)
0923 18:06:04.755516 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.755521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.761006 (+ 5485us) server_negotiation.cc:407] Connection header received
0923 18:06:04.761212 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.761215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.761270 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.761321 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.762321 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.763132 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.764389 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.764724 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.764824 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.766803 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.766808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.766811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.766843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.768324 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.768823 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.768827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.768828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.768919 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.769190 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.769193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.769195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.769435 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.769555 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.769836 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.769965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:06:04.780349 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.770243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.18.44.253:44899)
0923 18:06:04.770325 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.770329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.770594 (+ 265us) server_negotiation.cc:407] Connection header received
0923 18:06:04.770790 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.770793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.770848 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.770896 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.771893 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.772678 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.773846 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.774192 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.774263 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.776288 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.776292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.776295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.776324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.777966 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.778497 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.778501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.778503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.778600 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.779202 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.779206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.779209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.779493 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.779603 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.780051 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.780175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 18:06:04.781677 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.770948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.18.44.252:43589)
0923 18:06:04.771056 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.771060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.771939 (+ 879us) server_negotiation.cc:407] Connection header received
0923 18:06:04.772073 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.772076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.772111 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.772152 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.773129 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.773970 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.775189 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.776028 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.776120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.777877 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.777881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.777884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.777938 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.779909 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.780350 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.780354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.780356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.780440 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.780732 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.780736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.780738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.780959 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.781054 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.781401 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.781522 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":463,"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":24576}
I20260923 18:06:04.782747 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:04.755651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.18.44.252:43589)
0923 18:06:04.755949 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:04.755953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:04.771694 (+ 15741us) server_negotiation.cc:407] Connection header received
0923 18:06:04.771887 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:04.771891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:04.771942 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:04.771993 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:04.773350 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.774077 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.775541 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:04.775830 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:04.775925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:04.777877 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:04.777881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:04.777884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:04.777915 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:04.780104 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.780897 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.780900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.780902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.780998 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:04.781260 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:04.781263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:04.781265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:04.781461 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:04.781570 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:04.782476 (+ 906us) server_negotiation.cc:299] Negotiation successful
0923 18:06:04.782596 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":216,"spinlock_wait_cycles":896}
I20260923 18:06:05.736752 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:05.735755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:53095)
0923 18:06:05.736337 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:05.736574 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":85,"threads_started":1}
W20260923 18:06:05.737157 25821 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:44433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:05.738421 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:05.737873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:33577)
0923 18:06:05.738210 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:05.738309 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":82,"threads_started":1}
W20260923 18:06:05.738819 25957 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:44433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:05.739943 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:05.739193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:49907)
0923 18:06:05.739604 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:05.739834 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
W20260923 18:06:05.740594 25684 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:44433 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:06.253198 26027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0891315db754d67be66589360f29de3)
I20260923 18:06:06.253433 26028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0891315db754d67be66589360f29de3)
I20260923 18:06:06.253458 26027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:06.253729 26028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:06.254719 26027 leader_election.cc:288] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252:43589)
I20260923 18:06:06.254899 26028 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 88baa141f23e40caa57798777380a6b5 (127.18.44.253:44899)
I20260923 18:06:06.255635 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.254927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38572)
0923 18:06:06.255396 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.255514 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":101,"threads_started":1}
I20260923 18:06:06.255635 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.255104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38582)
0923 18:06:06.255448 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.255514 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":71,"threads_started":1}
W20260923 18:06:06.255849 25491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
W20260923 18:06:06.255972 25309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:06.257174 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.256878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38590)
0923 18:06:06.257031 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.257089 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 18:06:06.257205 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.256887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38588)
0923 18:06:06.257070 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.257111 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260923 18:06:06.257293 25309 leader_election.cc:334] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
W20260923 18:06:06.257293 25491 leader_election.cc:334] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:06.260673 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.255340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:43589 (local address 127.0.0.1:43594)
0923 18:06:06.255775 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.255791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:06.255905 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:06.256643 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:06.256646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:06.256662 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:06.256779 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:06.256786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.258003 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.258008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:06.259456 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:06.259469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.260274 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.260278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:06.260345 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:06.260363 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 18:06:06.260411 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 18:06:06.260487 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":51,"threads_started":1}
I20260923 18:06:06.260722 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.255668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.18.44.252:43589)
0923 18:06:06.255900 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:06.255904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:06.255920 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:06.256420 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:06.256423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:06.256463 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:06.256528 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:06.256907 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.257784 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.259606 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.260137 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.260250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:06.260394 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:06.260445 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 18:06:06.260542 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260923 18:06:06.260722 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.255159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:44899 (local address 127.0.0.1:46852)
0923 18:06:06.256036 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.256066 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:06.256228 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:06.256989 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:06.256992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:06.257008 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:06.257075 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:06.257080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.258212 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.258223 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:06.259471 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:06.259481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.260200 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.260204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:06.260312 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:06.260389 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 18:06:06.260465 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:06:06.260547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":45,"thread_start_us":131,"threads_started":1}
I20260923 18:06:06.260673 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.255537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.18.44.253:44899)
0923 18:06:06.256184 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:06.256214 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:06.256252 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:06:06.256529 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:06.256531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:06.256837 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:06.256907 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:06.257190 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.258081 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.259593 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:06.260073 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:06.260173 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:06.260244 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:06.260499 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 18:06:06.260563 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":46,"threads_started":1}
I20260923 18:06:06.261245 25513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88baa141f23e40caa57798777380a6b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" is_pre_election: true
I20260923 18:06:06.261554 25513 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88baa141f23e40caa57798777380a6b5 in term 1.
I20260923 18:06:06.261622 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88baa141f23e40caa57798777380a6b5" is_pre_election: true
I20260923 18:06:06.261930 25309 leader_election.cc:302] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d3e96220d474fad8ffc7ede7d039cb2, 88baa141f23e40caa57798777380a6b5; no voters: d0891315db754d67be66589360f29de3
I20260923 18:06:06.261986 25331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3e96220d474fad8ffc7ede7d039cb2 in term 1.
I20260923 18:06:06.262200 26027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 18:06:06.262274 26027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0891315db754d67be66589360f29de3)
I20260923 18:06:06.262315 26027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:06:06.262454 25488 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d3e96220d474fad8ffc7ede7d039cb2, 88baa141f23e40caa57798777380a6b5; no voters: d0891315db754d67be66589360f29de3
I20260923 18:06:06.262761 26028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 18:06:06.262827 26028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0891315db754d67be66589360f29de3)
I20260923 18:06:06.262856 26028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:06:06.263828 26027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:06.264268 26027 leader_election.cc:288] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 election: Requested vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 0d3e96220d474fad8ffc7ede7d039cb2 (127.18.44.252:43589)
I20260923 18:06:06.264238 26028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:06.264746 26028 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 election: Requested vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 88baa141f23e40caa57798777380a6b5 (127.18.44.253:44899)
I20260923 18:06:06.265081 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.264573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38606)
0923 18:06:06.264906 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.264979 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260923 18:06:06.265193 25513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88baa141f23e40caa57798777380a6b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d3e96220d474fad8ffc7ede7d039cb2"
I20260923 18:06:06.265195 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88baa141f23e40caa57798777380a6b5"
I20260923 18:06:06.265298 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.264949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38618)
0923 18:06:06.265143 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.265209 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 18:06:06.265390 25331 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d3e96220d474fad8ffc7ede7d039cb2 in current term 2: Already voted for candidate 88baa141f23e40caa57798777380a6b5 in this term.
I20260923 18:06:06.265393 25513 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88baa141f23e40caa57798777380a6b5 in current term 2: Already voted for candidate 0d3e96220d474fad8ffc7ede7d039cb2 in this term.
I20260923 18:06:06.266116 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.265912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38632)
0923 18:06:06.266014 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.266062 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 18:06:06.266240 25491 leader_election.cc:334] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:06.266319 25491 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d3e96220d474fad8ffc7ede7d039cb2; no voters: 88baa141f23e40caa57798777380a6b5, d0891315db754d67be66589360f29de3
I20260923 18:06:06.266451 26028 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 18:06:06.266810 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.266134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38636)
0923 18:06:06.266696 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.266747 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
W20260923 18:06:06.266952 25309 leader_election.cc:334] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:06.267038 25309 leader_election.cc:302] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88baa141f23e40caa57798777380a6b5; no voters: 0d3e96220d474fad8ffc7ede7d039cb2, d0891315db754d67be66589360f29de3
I20260923 18:06:06.267210 26027 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 18:06:06.738591 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.737910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:60275)
0923 18:06:06.738378 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.738458 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":117,"threads_started":1}
I20260923 18:06:06.740435 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.739664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:35365)
0923 18:06:06.740241 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.740327 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":153,"threads_started":1}
I20260923 18:06:06.741868 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:06.741322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:48355)
0923 18:06:06.741672 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:06.741745 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260923 18:06:07.740376 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.739676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:37731)
0923 18:06:07.739991 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.740192 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
I20260923 18:06:07.742172 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.741522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:47931)
0923 18:06:07.741941 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.742026 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":125,"threads_started":1}
I20260923 18:06:07.743223 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.742691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:42813)
0923 18:06:07.743032 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.743113 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260923 18:06:07.845204 26043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:07.845424 26043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:07.845968 26043 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 88baa141f23e40caa57798777380a6b5 (127.18.44.253:44899)
I20260923 18:06:07.846647 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88baa141f23e40caa57798777380a6b5" is_pre_election: true
I20260923 18:06:07.846798 25331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3e96220d474fad8ffc7ede7d039cb2 in term 2.
I20260923 18:06:07.846938 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.846185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38880)
0923 18:06:07.846570 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.846640 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":93,"threads_started":1}
I20260923 18:06:07.847380 25488 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3e96220d474fad8ffc7ede7d039cb2, 88baa141f23e40caa57798777380a6b5; no voters:
I20260923 18:06:07.847744 26043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 18:06:07.847874 26043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:07.847920 26043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:06:07.848513 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.848039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38894)
0923 18:06:07.848244 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.848335 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260923 18:06:07.848714 25491 leader_election.cc:334] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:07.849277 26043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:07.849696 26043 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 election: Requested vote from peers d0891315db754d67be66589360f29de3 (127.18.44.254:44433), 88baa141f23e40caa57798777380a6b5 (127.18.44.253:44899)
I20260923 18:06:07.850389 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.850035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38896)
0923 18:06:07.850204 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.850289 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 18:06:07.850417 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88baa141f23e40caa57798777380a6b5"
I20260923 18:06:07.850566 25331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:06:07.851424 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.851085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38898)
0923 18:06:07.851221 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.851349 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 18:06:07.851617 25491 leader_election.cc:334] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
I20260923 18:06:07.851989 25331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3e96220d474fad8ffc7ede7d039cb2 in term 3.
I20260923 18:06:07.852396 25488 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d3e96220d474fad8ffc7ede7d039cb2, 88baa141f23e40caa57798777380a6b5; no voters: d0891315db754d67be66589360f29de3
I20260923 18:06:07.852594 26043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 18:06:07.852819 26043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [term 3 LEADER]: Becoming Leader. State: Replica: 0d3e96220d474fad8ffc7ede7d039cb2, State: Running, Role: LEADER
I20260923 18:06:07.853091 26043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } }
I20260923 18:06:07.854952 26046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3e96220d474fad8ffc7ede7d039cb2. Latest consensus state: current_term: 3 leader_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:07.855203 26046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:07.855813 26048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:07.858357 26048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:07.858951 26048 catalog_manager.cc:1259] Loaded cluster ID: ba34ba7ebde84ca49875e1e917af0539
I20260923 18:06:07.859006 26048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:07.859750 26048 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:07.860146 26048 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Loaded TSK: 0
I20260923 18:06:07.860864 26048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:06:07.877460 25503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:07.881063 25503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:07.885007 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.884734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38910)
0923 18:06:07.884865 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.884926 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 18:06:07.885427 25331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [term 3 FOLLOWER]: Refusing update from remote peer 0d3e96220d474fad8ffc7ede7d039cb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 18:06:07.885999 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.885547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.0.0.1:38916)
0923 18:06:07.885768 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.885887 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 18:06:07.886209 25491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 -> Peer d0891315db754d67be66589360f29de3 (127.18.44.254:44433): Couldn't send request to peer d0891315db754d67be66589360f29de3. Status: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 18:06:07.886269 26046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 18:06:07.888685 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3e96220d474fad8ffc7ede7d039cb2. Latest consensus state: current_term: 3 leader_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:07.888988 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:07.891080 26046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:07.891345 26046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:07.891384 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0891315db754d67be66589360f29de3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44433 } } peers { permanent_uuid: "88baa141f23e40caa57798777380a6b5" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 44899 } attrs { promote: false } } peers { permanent_uuid: "0d3e96220d474fad8ffc7ede7d039cb2" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 43589 } attrs { promote: false } } }
I20260923 18:06:07.891609 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88baa141f23e40caa57798777380a6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:06:07.891664 26051 mvcc.cc:205] Tried to move back new op lower bound from 7332605001250775040 to 7332605001129107456. Current Snapshot: MvccSnapshot[applied={T|T < 7332605001250775040}]
I20260923 18:06:07.907233 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.902069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:46483 (local address 127.0.0.1:41104)
0923 18:06:07.902420 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.902435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:07.902518 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:07.903549 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:07.903552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:07.903568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:07.903780 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.903791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.904668 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.904671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.905846 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.905856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.906839 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.906842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.906920 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.906947 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:06:07.906989 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 18:06:07.907074 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":178,"thread_start_us":131,"threads_started":1}
I20260923 18:06:07.907775 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.901865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46033 (local address 127.0.0.1:57086)
0923 18:06:07.902017 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.902032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:07.902123 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:07.903107 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:07.903110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:07.903128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:07.903717 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.903725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.905116 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.905119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.906363 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.906372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.907362 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.907366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.907459 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.907485 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:06:07.907547 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 18:06:07.907660 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":70}
I20260923 18:06:07.907891 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.902974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.18.44.193:46483)
0923 18:06:07.903292 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:07.903297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:07.903310 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:06:07.903375 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:07.903378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:07.903422 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:07.903895 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:07.904013 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.904547 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.906010 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.906675 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.907534 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.907628 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:07.907669 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 18:06:07.907754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":145,"threads_started":1}
I20260923 18:06:07.907902 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.902063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.18.44.195:46033)
0923 18:06:07.902744 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:07.902748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:07.902763 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:06:07.902812 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:07.902815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:07.902868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:07.903024 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:07.903851 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.904915 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.906543 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.907177 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.907424 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.907534 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:07.907585 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 18:06:07.907692 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":152,"threads_started":1}
I20260923 18:06:07.909067 25890 tablet_service.cc:1467] Processing CreateTablet for tablet f1299bb3fcef434687662de6a4d013fa (DEFAULT_TABLE table=test-table [id=9b94e9581dc44262a56d297310324d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:07.909111 25617 tablet_service.cc:1467] Processing CreateTablet for tablet f1299bb3fcef434687662de6a4d013fa (DEFAULT_TABLE table=test-table [id=9b94e9581dc44262a56d297310324d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:07.909473 25890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1299bb3fcef434687662de6a4d013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:07.909584 25617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1299bb3fcef434687662de6a4d013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:07.912052 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.903437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:45397 (local address 127.0.0.1:49350)
0923 18:06:07.905190 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.905209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:07.905339 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:07.905633 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:07.905636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:07.905656 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:07.905803 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.905810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.906830 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.906833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.908080 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:07.908093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.911487 (+ 3394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.911493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:07.911637 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.911671 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 18:06:07.911782 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 18:06:07.911879 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"mutex_wait_us":5,"thread_start_us":130,"threads_started":1}
I20260923 18:06:07.912603 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.903673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.18.44.194:45397)
0923 18:06:07.904055 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:07.904059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:07.905320 (+ 1261us) server_negotiation.cc:407] Connection header received
0923 18:06:07.905453 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:07.905455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:07.905498 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:07.905595 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:07.905931 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.906710 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.908315 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:07.908960 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:07.909112 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:07.909177 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:07.911875 (+ 2698us) server_negotiation.cc:299] Negotiation successful
0923 18:06:07.912087 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":112,"threads_started":1}
I20260923 18:06:07.913197 26062 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: Bootstrap starting.
I20260923 18:06:07.913367 26061 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Bootstrap starting.
I20260923 18:06:07.913403 25754 tablet_service.cc:1467] Processing CreateTablet for tablet f1299bb3fcef434687662de6a4d013fa (DEFAULT_TABLE table=test-table [id=9b94e9581dc44262a56d297310324d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:07.913739 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1299bb3fcef434687662de6a4d013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:07.914260 26062 tablet_bootstrap.cc:654] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:07.914949 26061 tablet_bootstrap.cc:654] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:07.916096 26062 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: No bootstrap required, opened a new log
I20260923 18:06:07.916229 26062 ts_tablet_manager.cc:1397] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:06:07.916734 26062 raft_consensus.cc:348] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.916859 26062 raft_consensus.cc:374] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:07.916891 26062 raft_consensus.cc:729] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cd43e195e24ea2af9092dcf35ed533, State: Initialized, Role: FOLLOWER
I20260923 18:06:07.917032 26062 consensus_queue.cc:260] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.917459 26062 ts_tablet_manager.cc:1428] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:07.918412 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.918127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:52745)
0923 18:06:07.918252 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.918324 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 18:06:07.918512 26066 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Bootstrap starting.
I20260923 18:06:07.919292 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.918983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:46505)
0923 18:06:07.919137 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.919186 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 18:06:07.919885 26066 tablet_bootstrap.cc:654] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:07.921728 25683 heartbeater.cc:503] Master 127.18.44.252:43589 was elected leader, sending a full tablet report...
I20260923 18:06:07.921888 26061 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: No bootstrap required, opened a new log
I20260923 18:06:07.922050 26061 ts_tablet_manager.cc:1397] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260923 18:06:07.922593 26061 raft_consensus.cc:348] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.922749 26061 raft_consensus.cc:374] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:07.922799 26061 raft_consensus.cc:729] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133a7fcb4fe45a08470cc277e8bca73, State: Initialized, Role: FOLLOWER
I20260923 18:06:07.922989 26061 consensus_queue.cc:260] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.923178 26066 tablet_bootstrap.cc:492] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: No bootstrap required, opened a new log
I20260923 18:06:07.923370 26066 ts_tablet_manager.cc:1397] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 18:06:07.923549 26061 ts_tablet_manager.cc:1428] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 18:06:07.923983 26066 raft_consensus.cc:348] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.924120 26066 raft_consensus.cc:374] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:07.924151 26066 raft_consensus.cc:729] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9aab5e9576496e809caf334432579b, State: Initialized, Role: FOLLOWER
I20260923 18:06:07.924289 26066 consensus_queue.cc:260] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.924500 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.924076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:54329)
0923 18:06:07.924273 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.924374 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 18:06:07.924788 26066 ts_tablet_manager.cc:1428] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:07.926064 25956 heartbeater.cc:503] Master 127.18.44.252:43589 was elected leader, sending a full tablet report...
I20260923 18:06:07.926782 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.925666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:53819)
0923 18:06:07.926570 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.926664 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723}
I20260923 18:06:07.930794 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.930435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:50939)
0923 18:06:07.930601 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.930674 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 18:06:07.931963 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.931734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:49945)
0923 18:06:07.931833 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.931884 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 18:06:07.932029 25820 heartbeater.cc:503] Master 127.18.44.252:43589 was elected leader, sending a full tablet report...
I20260923 18:06:07.978160 26065 raft_consensus.cc:482] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:07.978420 26065 raft_consensus.cc:504] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.978930 26065 leader_election.cc:288] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033), bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:07.979584 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1299bb3fcef434687662de6a4d013fa" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9aab5e9576496e809caf334432579b" is_pre_election: true
I20260923 18:06:07.979647 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1299bb3fcef434687662de6a4d013fa" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133a7fcb4fe45a08470cc277e8bca73" is_pre_election: true
I20260923 18:06:07.979755 25774 raft_consensus.cc:2454] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 0.
I20260923 18:06:07.979897 25910 raft_consensus.cc:2454] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 0.
I20260923 18:06:07.980189 25573 leader_election.cc:302] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9aab5e9576496e809caf334432579b, c3cd43e195e24ea2af9092dcf35ed533; no voters:
I20260923 18:06:07.980443 26065 raft_consensus.cc:2781] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:07.980522 26065 raft_consensus.cc:482] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:07.980551 26065 raft_consensus.cc:3037] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:07.982007 26065 raft_consensus.cc:504] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.982503 26065 leader_election.cc:288] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 election: Requested vote from peers 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033), bc9aab5e9576496e809caf334432579b (127.18.44.194:45397)
I20260923 18:06:07.983349 25774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1299bb3fcef434687662de6a4d013fa" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9aab5e9576496e809caf334432579b"
I20260923 18:06:07.983530 25774 raft_consensus.cc:3037] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:07.983474 25910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1299bb3fcef434687662de6a4d013fa" candidate_uuid: "c3cd43e195e24ea2af9092dcf35ed533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133a7fcb4fe45a08470cc277e8bca73"
I20260923 18:06:07.983732 25910 raft_consensus.cc:3037] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:07.984830 25774 raft_consensus.cc:2454] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 1.
I20260923 18:06:07.984861 25910 raft_consensus.cc:2454] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3cd43e195e24ea2af9092dcf35ed533 in term 1.
I20260923 18:06:07.985275 25573 leader_election.cc:302] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9aab5e9576496e809caf334432579b, c3cd43e195e24ea2af9092dcf35ed533; no voters:
I20260923 18:06:07.985581 26065 raft_consensus.cc:2781] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:07.985872 26065 raft_consensus.cc:686] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 1 LEADER]: Becoming Leader. State: Replica: c3cd43e195e24ea2af9092dcf35ed533, State: Running, Role: LEADER
I20260923 18:06:07.986050 26065 consensus_queue.cc:237] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } }
I20260923 18:06:07.986900 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:07.986553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:52067)
0923 18:06:07.986682 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:07.986746 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 18:06:07.987978 25504 catalog_manager.cc:5495] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "c3cd43e195e24ea2af9092dcf35ed533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3cd43e195e24ea2af9092dcf35ed533" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 46483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 } health_report { overall_health: UNKNOWN } } }
I20260923 18:06:08.018384 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.014350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.18.44.193:46483)
0923 18:06:08.014533 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:08.014540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:08.014612 (+ 72us) server_negotiation.cc:407] Connection header received
0923 18:06:08.014691 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:08.014695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:08.014765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:08.014864 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:08.015299 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:08.015919 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:08.017028 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:08.017292 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:08.017437 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:08.017817 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:08.017939 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:08.018024 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:08.018099 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 18:06:08.018175 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:06:08.021198 25774 raft_consensus.cc:1261] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Refusing update from remote peer c3cd43e195e24ea2af9092dcf35ed533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:08.021481 25910 raft_consensus.cc:1261] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Refusing update from remote peer c3cd43e195e24ea2af9092dcf35ed533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:08.022063 26065 consensus_queue.cc:1035] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9aab5e9576496e809caf334432579b" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:08.022372 26070 consensus_queue.cc:1035] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133a7fcb4fe45a08470cc277e8bca73" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:08.025034 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.024638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:51039)
0923 18:06:08.024826 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.024897 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 18:06:08.025774 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.025431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:58067)
0923 18:06:08.025585 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.025674 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 18:06:08.030048 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.029668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:57863)
0923 18:06:08.029867 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.029934 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260923 18:06:08.031848 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.031387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.195:47139)
0923 18:06:08.031668 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.031744 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260923 18:06:08.033813 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.033188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.194:37099)
0923 18:06:08.033631 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.033713 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
I20260923 18:06:08.034514 25502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:08.034683 25502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:08.035352 26074 mvcc.cc:205] Tried to move back new op lower bound from 7332605001809674240 to 7332605001672507392. Current Snapshot: MvccSnapshot[applied={T|T < 7332605001809674240}]
I20260923 18:06:08.036484 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.036134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44433 (local address 127.18.44.193:57781)
0923 18:06:08.036288 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.036364 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 18:06:08.040035 25502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d3e96220d474fad8ffc7ede7d039cb2: Sending DeleteTablet for 3 replicas of tablet f1299bb3fcef434687662de6a4d013fa
I20260923 18:06:08.040764 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet f1299bb3fcef434687662de6a4d013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:08 UTC) from {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:49350
I20260923 18:06:08.041055 25617 tablet_service.cc:1514] Processing DeleteTablet for tablet f1299bb3fcef434687662de6a4d013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:08 UTC) from {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:41104
I20260923 18:06:08.041357 26082 tablet_replica.cc:330] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: stopping tablet replica
I20260923 18:06:08.041440 25753 tablet_service.cc:1514] Processing DeleteTablet for tablet f1299bb3fcef434687662de6a4d013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:08 UTC) from {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:49350
I20260923 18:06:08.041530 26082 raft_consensus.cc:2229] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:08.041711 26082 raft_consensus.cc:2258] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:08.041745 26083 tablet_replica.cc:330] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: stopping tablet replica
I20260923 18:06:08.042035 26083 raft_consensus.cc:2229] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:08.042261 25890 tablet_service.cc:1514] Processing DeleteTablet for tablet f1299bb3fcef434687662de6a4d013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:08 UTC) from {username='oryx', principal='oryx/127.18.44.252@KRBTEST.COM'} at 127.0.0.1:57086
I20260923 18:06:08.042574 26083 raft_consensus.cc:2258] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:08.042970 26084 tablet_replica.cc:330] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: stopping tablet replica
I20260923 18:06:08.042968 26082 ts_tablet_manager.cc:1905] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:08.043156 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25466
I20260923 18:06:08.043563 26083 ts_tablet_manager.cc:1905] T f1299bb3fcef434687662de6a4d013fa P c3cd43e195e24ea2af9092dcf35ed533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:08.045077 26084 raft_consensus.cc:2229] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:08.045398 26084 raft_consensus.cc:2258] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:08.046382 26084 ts_tablet_manager.cc:1905] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:08.046811 26082 ts_tablet_manager.cc:1918] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:08.046909 26082 log.cc:1196] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/f1299bb3fcef434687662de6a4d013fa
I20260923 18:06:08.047101 26082 ts_tablet_manager.cc:1939] T f1299bb3fcef434687662de6a4d013fa P bc9aab5e9576496e809caf334432579b: Deleting consensus metadata
I20260923 18:06:08.048557 25491 catalog_manager.cc:4841] TS bc9aab5e9576496e809caf334432579b (127.18.44.194:45397): tablet f1299bb3fcef434687662de6a4d013fa (table test-table [id=9b94e9581dc44262a56d297310324d4c]) successfully deleted
W20260923 18:06:08.048926 25491 catalog_manager.cc:4808] TS bc9aab5e9576496e809caf334432579b (127.18.44.194:45397): delete failed for tablet f1299bb3fcef434687662de6a4d013fa because the tablet was not found. No further retry: Not found: Tablet not found: f1299bb3fcef434687662de6a4d013fa
I20260923 18:06:08.050006 26084 ts_tablet_manager.cc:1918] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:08.050128 26084 log.cc:1196] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/f1299bb3fcef434687662de6a4d013fa
I20260923 18:06:08.050383 26084 ts_tablet_manager.cc:1939] T f1299bb3fcef434687662de6a4d013fa P 7133a7fcb4fe45a08470cc277e8bca73: Deleting consensus metadata
I20260923 18:06:08.051731 25488 catalog_manager.cc:4841] TS 7133a7fcb4fe45a08470cc277e8bca73 (127.18.44.195:46033): tablet f1299bb3fcef434687662de6a4d013fa (table test-table [id=9b94e9581dc44262a56d297310324d4c]) successfully deleted
I20260923 18:06:08.054318 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25691
I20260923 18:06:08.054579 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:08.054293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:46483 (local address 127.0.0.1:41114)
0923 18:06:08.054416 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:08.054470 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 18:06:08.055518 25489 catalog_manager.cc:4584] TS c3cd43e195e24ea2af9092dcf35ed533 (127.18.44.193:46483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1299bb3fcef434687662de6a4d013fa: Network error: Client connection negotiation failed: client connection to 127.18.44.193:46483: connect: Connection refused (error 111)
I20260923 18:06:08.061298 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25827
I20260923 18:06:08.069291 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25183
I20260923 18:06:08.075903 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 25285
2026-09-23T18:06:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7463 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:08 dist-test-slave-rvjj krb5kdc[26087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:08 dist-test-slave-rvjj krb5kdc[26087](info): set up 2 sockets
Sep 23 18:06:08 dist-test-slave-rvjj krb5kdc[26087](info): commencing operation
krb5kdc: starting...
W20260923 18:06:09.108999 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:09 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab.
Entry for principal oryx/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab.
I20260923 18:06:09.259739 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37739
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab
--principal=oryx/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:09.402755 26100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:09.403226 26100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:09.403368 26100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:09.408381 26100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:09.408501 26100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:06:09.408528 26100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:09.408556 26100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:09.408581 26100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:09.408600 26100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:09.415838 26100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37739
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.254.keytab
--principal=oryx/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:09.417665 26100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:09.419411 26100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:09.433578 26105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:09.433578 26106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:09.433642 26108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:09.434250 26100 server_base.cc:1034] running on GCE node
I20260923 18:06:09.434996 26100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:09.436406 26100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:09.437670 26100 hybrid_clock.cc:648] HybridClock initialized: now 1790186769437651 us; error 61 us; skew 500 ppm
Sep 23 18:06:09 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:09.441707 26100 init.cc:377] Logged in from keytab as oryx/127.18.44.254@KRBTEST.COM (short username oryx)
I20260923 18:06:09.442983 26100 webserver.cc:460] Webserver started at http://127.18.44.254:33937/ using document root <none> and password file <none>
I20260923 18:06:09.443943 26100 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:09.444042 26100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:09.444274 26100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:09.447361 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "1855a2c327a246efac22606e31b55653"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "283f68223b76174eb6039f276c118cb5"
server_key_iv: "f0e4df78758d5bec3d2930371b9e8279"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.448422 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "1855a2c327a246efac22606e31b55653"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "283f68223b76174eb6039f276c118cb5"
server_key_iv: "f0e4df78758d5bec3d2930371b9e8279"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.453637 26100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:06:09.457592 26115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:09.459861 26100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:06:09.460078 26100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "1855a2c327a246efac22606e31b55653"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "283f68223b76174eb6039f276c118cb5"
server_key_iv: "f0e4df78758d5bec3d2930371b9e8279"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.460232 26100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:09.498262 26100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:09.502501 26100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:09.502763 26100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:09.514138 26100 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37739
I20260923 18:06:09.514191 26177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37739 every 8 connection(s)
I20260923 18:06:09.515357 26100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:09.519778 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26100
I20260923 18:06:09.519841 26178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:09.519925 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:09.520288 18611 external_mini_cluster.cc:1428] Setting key 02154208115c3d649c29b50d463ba69f
I20260923 18:06:09.532299 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Bootstrap starting.
I20260923 18:06:09.536155 26178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:09.537595 26178 log.cc:824] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:06:09 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:09.540861 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: No bootstrap required, opened a new log
I20260923 18:06:09.544875 26178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } }
I20260923 18:06:09.545259 26178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:09.545310 26178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1855a2c327a246efac22606e31b55653, State: Initialized, Role: FOLLOWER
I20260923 18:06:09.546170 26178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } }
I20260923 18:06:09.546375 26178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:09.546439 26178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:09.546540 26178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:09.548307 26178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } }
I20260923 18:06:09.548916 26178 leader_election.cc:302] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1855a2c327a246efac22606e31b55653; no voters:
I20260923 18:06:09.549343 26178 leader_election.cc:288] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:09.549543 26183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:09.549976 26183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [term 1 LEADER]: Becoming Leader. State: Replica: 1855a2c327a246efac22606e31b55653, State: Running, Role: LEADER
I20260923 18:06:09.550402 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:09.522204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.18.44.254:37739)
0923 18:06:09.522696 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:09.522713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:09.522753 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:06:09.524244 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:09.524277 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:09.524732 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:09.525219 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:09.526668 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.528268 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.530290 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.531076 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.531370 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:09.540722 (+ 9352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:09.540748 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:09.540776 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:09.540835 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:09.544335 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.545184 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.545192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.545199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.545542 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.546153 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.546156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.546158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.547248 (+ 1090us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:09.547640 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:09.548360 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 18:06:09.548916 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260923 18:06:09.550454 26183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } }
I20260923 18:06:09.550990 26178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:09.552834 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1855a2c327a246efac22606e31b55653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } } }
I20260923 18:06:09.553071 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:09.553504 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1855a2c327a246efac22606e31b55653. Latest consensus state: current_term: 1 leader_uuid: "1855a2c327a246efac22606e31b55653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1855a2c327a246efac22606e31b55653" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37739 } } }
I20260923 18:06:09.553854 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653 [sys.catalog]: This master's current role is: LEADER
W20260923 18:06:09.557122 26198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:06:09.557243 26198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:06:09.557356 26197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:09.558280 26197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:09.566864 26197 catalog_manager.cc:1347] Generated new cluster ID: b63b5f3adbdb4cf5b7a86432b555528a
I20260923 18:06:09.566967 26197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:09.580845 26197 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:09.582646 26197 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:09.592865 26197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Generated new TSK 0
I20260923 18:06:09.593730 26197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
I20260923 18:06:09.638904 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37739
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:09.822784 26204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:09.823227 26204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:09.823348 26204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:09.829355 26204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:09.829485 26204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:09.829537 26204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:06:09.829577 26204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:09.839071 26204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:09.840390 26204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:09.842072 26204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:09.856806 26209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:09.856809 26212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:09.856889 26210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:09.857365 26204 server_base.cc:1034] running on GCE node
I20260923 18:06:09.858110 26204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:09.859100 26204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:09.860491 26204 hybrid_clock.cc:648] HybridClock initialized: now 1790186769860447 us; error 68 us; skew 500 ppm
Sep 23 18:06:09 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:09.864748 26204 init.cc:377] Logged in from keytab as oryx/127.18.44.193@KRBTEST.COM (short username oryx)
I20260923 18:06:09.866180 26204 webserver.cc:460] Webserver started at http://127.18.44.193:37427/ using document root <none> and password file <none>
I20260923 18:06:09.866950 26204 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:09.867043 26204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:09.867344 26204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:09.870672 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "13b87f127ec442fe92d4eb83a4effad1"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "a11a4475b0f698e8d88a60e145ed87b4"
server_key_iv: "1d94d03e387fcf6113264479a3925bb7"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.871647 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "13b87f127ec442fe92d4eb83a4effad1"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "a11a4475b0f698e8d88a60e145ed87b4"
server_key_iv: "1d94d03e387fcf6113264479a3925bb7"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.877817 26204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 18:06:09.882611 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:09.884917 26204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:09.885136 26204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "13b87f127ec442fe92d4eb83a4effad1"
format_stamp: "Formatted at 2026-09-23 18:06:09 on dist-test-slave-rvjj"
server_key: "a11a4475b0f698e8d88a60e145ed87b4"
server_key_iv: "1d94d03e387fcf6113264479a3925bb7"
server_key_version: "encryptionkey@0"
I20260923 18:06:09.885353 26204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:09.898000 26204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:09.903203 26204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:09.903551 26204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:09.906159 26204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:09.906250 26204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:09.906307 26204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:09.906322 26204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:06:09 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:09.920522 26204 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:45115
I20260923 18:06:09.920651 26334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:45115 every 8 connection(s)
I20260923 18:06:09.921948 26204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:09.922724 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:09.907391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:42815 (local address 127.18.44.254:37739)
0923 18:06:09.907545 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:09.907549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:09.909243 (+ 1694us) server_negotiation.cc:407] Connection header received
0923 18:06:09.910565 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:09.910569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:09.910636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:09.910717 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:09.912293 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.912832 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.914143 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.914422 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.914524 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:09.918478 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:09.918482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:09.918484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:09.918515 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:09.920043 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.920795 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.920799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.920801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.920886 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.921329 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.921333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.921334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.921553 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:09.921669 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:09.922399 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 18:06:09.922551 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:06:09.923888 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:09.908205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.193:42815)
0923 18:06:09.909014 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:09.909080 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:09.910322 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:09.910952 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:09.910963 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:09.911488 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:09.912119 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:09.912135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.912972 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.912975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:09.913947 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:09.913961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.914615 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.914620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:09.914804 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:09.915520 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:09.915539 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:09.918237 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:09.920220 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:09.920228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:09.920250 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:09.920646 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:09.920977 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:09.920980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:09.920982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:09.921142 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:09.921691 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:09.921699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:09.922096 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 18:06:09.922324 (+ 228us) client_negotiation.cc:240] Negotiation successful
0923 18:06:09.922702 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":225,"threads_started":1}
I20260923 18:06:09.925989 26335 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37739
I20260923 18:06:09.926256 26335 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:09.927395 26335 heartbeater.cc:511] Master 127.18.44.254:37739 requested a full tablet report, sending...
I20260923 18:06:09.929795 26128 ts_manager.cc:194] Registered new tserver with Master: 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115)
I20260923 18:06:09.930104 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26204
I20260923 18:06:09.930207 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:09.930541 18611 external_mini_cluster.cc:1428] Setting key 8b306e5f9adcb2c2f2a04acb6fc7ad9e
I20260923 18:06:09.931128 26128 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:42815
I20260923 18:06:09.939939 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:09.928886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:32841 (local address 127.18.44.254:37739)
0923 18:06:09.929052 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:09.929058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:09.929527 (+ 469us) server_negotiation.cc:407] Connection header received
0923 18:06:09.929625 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:09.929628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:09.929673 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:09.929761 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:09.930355 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.931336 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.932624 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.932971 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.933094 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:09.934879 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:09.934883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:09.934886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:09.934944 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:09.937117 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.938372 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.938394 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.938395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.938517 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:09.938908 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:09.938911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:09.938913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:09.939152 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:09.939236 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:09.939690 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 18:06:09.939810 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:06:09.940145 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:09.928732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.193:32841)
0923 18:06:09.929339 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:09.929357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:09.929528 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:09.929801 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:09.929804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:09.930047 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:09.930214 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:09.930222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.931547 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.931551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:09.932452 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:09.932470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:09.933148 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:09.933153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:09.933274 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:09.934060 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:09.934075 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:09.934644 (+ 569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:09.937478 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:09.937482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:09.937484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:09.937892 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:09.938621 (+ 729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:09.938625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:09.938627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:09.938758 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:09.939289 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:09.939292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:09.939416 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 18:06:09.939535 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 18:06:09.939818 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":181,"threads_started":1}
WARNING: no policy specified for oryx/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
I20260923 18:06:09.971728 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37739
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:10.143209 26344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:10.143800 26344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:10.143864 26344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:10.149005 26344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:10.149133 26344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:10.149181 26344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:06:10.149220 26344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:10.156556 26344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:10.158155 26344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:10.159646 26344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:10.173120 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:10.173156 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:10.173462 26344 server_base.cc:1034] running on GCE node
W20260923 18:06:10.173156 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:10.174247 26344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:10.175029 26344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:10.176275 26344 hybrid_clock.cc:648] HybridClock initialized: now 1790186770176239 us; error 63 us; skew 500 ppm
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:10.179912 26344 init.cc:377] Logged in from keytab as oryx/127.18.44.194@KRBTEST.COM (short username oryx)
I20260923 18:06:10.181584 26344 webserver.cc:460] Webserver started at http://127.18.44.194:46485/ using document root <none> and password file <none>
I20260923 18:06:10.183009 26344 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:10.183137 26344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:10.183478 26344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:10.186365 26344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "b64f6ec176cb43a9853613f95c33a43f"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "c35bbf4b85d8fe7c7ec09bea3208a984"
server_key_iv: "70b93a3e8572d665c78d5b7859b856b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.187135 26344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "b64f6ec176cb43a9853613f95c33a43f"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "c35bbf4b85d8fe7c7ec09bea3208a984"
server_key_iv: "70b93a3e8572d665c78d5b7859b856b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.192734 26344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 18:06:10.197111 26359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:10.199443 26344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:06:10.199704 26344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "b64f6ec176cb43a9853613f95c33a43f"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "c35bbf4b85d8fe7c7ec09bea3208a984"
server_key_iv: "70b93a3e8572d665c78d5b7859b856b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.199885 26344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:10.217623 26344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:10.222798 26344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:10.223086 26344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:10.225203 26344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:10.225275 26344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:10.225339 26344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:10.225368 26344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:10.240338 26344 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39871
I20260923 18:06:10.240381 26474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39871 every 8 connection(s)
I20260923 18:06:10.241842 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.226384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:55349 (local address 127.18.44.254:37739)
0923 18:06:10.226536 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.226542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.227723 (+ 1181us) server_negotiation.cc:407] Connection header received
0923 18:06:10.228823 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.228826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.228885 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.228963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.230582 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.231138 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.232521 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.232749 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.232841 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.236612 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:10.236616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:10.236618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:10.236654 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:10.238406 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.239476 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.239483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.239485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.239703 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.240414 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.240419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.240421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.240726 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:10.240838 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.241497 (+ 659us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.241664 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:06:10.241850 26344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:10.242260 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26344
I20260923 18:06:10.242415 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:10.242946 18611 external_mini_cluster.cc:1428] Setting key e9719561aff2d45654eab1c0182283ae
I20260923 18:06:10.243124 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.226901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.194:55349)
0923 18:06:10.227532 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.227589 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.228575 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.229316 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.229326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.229743 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:10.230423 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.230441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.231309 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.231313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.232389 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.232401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.232910 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.232915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.233093 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.233982 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:10.234006 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:10.236365 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:10.238606 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.238615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.238641 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.239200 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.239890 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.239894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.239897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.240252 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.240858 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:10.240866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:10.241213 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.241406 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.241848 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":160,"threads_started":1}
I20260923 18:06:10.246160 26475 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37739
I20260923 18:06:10.246500 26475 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:10.247542 26475 heartbeater.cc:511] Master 127.18.44.254:37739 requested a full tablet report, sending...
I20260923 18:06:10.249485 26128 ts_manager.cc:194] Registered new tserver with Master: b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871)
I20260923 18:06:10.250409 26128 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:55349
WARNING: no policy specified for oryx/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.195@KRBTEST.COM" created.
I20260923 18:06:10.259193 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.250113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:53703 (local address 127.18.44.254:37739)
0923 18:06:10.250251 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.250256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.250278 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:06:10.250341 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.250345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.250399 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.250512 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.251217 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.251982 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.253169 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.253409 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.253523 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.255197 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:10.255201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:10.255203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:10.255300 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:10.257063 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.257698 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.257703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.257705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.257803 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.258095 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.258098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.258100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.258418 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:10.258531 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.258829 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.258976 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:06:10.259207 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.249583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.194:53703)
0923 18:06:10.250066 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.250087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.250243 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.250583 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.250586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.250927 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:10.251077 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.251085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.252140 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.252144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.253036 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.253049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.253557 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.253562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.253662 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.254381 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:10.254398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:10.254980 (+ 582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:10.257216 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.257221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.257224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.257582 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.257898 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.257902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.257904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.258001 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.258524 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:10.258528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:10.258653 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.258751 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.258968 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":167,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
I20260923 18:06:10.282541 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37739
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:10.407784 26484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:10.408106 26484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:10.408152 26484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:10.412971 26484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:10.413077 26484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:10.413122 26484 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:06:10.413156 26484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:10.419068 26484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.44.254:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:10.420308 26484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:10.421540 26484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:10.434544 26492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:10.434608 26489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:10.434949 26484 server_base.cc:1034] running on GCE node
W20260923 18:06:10.434548 26490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:10.435789 26484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:10.436821 26484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:10.438079 26484 hybrid_clock.cc:648] HybridClock initialized: now 1790186770438031 us; error 80 us; skew 500 ppm
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:10.442852 26484 init.cc:377] Logged in from keytab as oryx/127.18.44.195@KRBTEST.COM (short username oryx)
I20260923 18:06:10.444475 26484 webserver.cc:460] Webserver started at http://127.18.44.195:35151/ using document root <none> and password file <none>
I20260923 18:06:10.445338 26484 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:10.445443 26484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:10.445716 26484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:10.448858 26484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "42a379d2815b4fe0a91f1b37628819b2"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "a192e0f9ba23d7378bd49bc0f915c03b"
server_key_iv: "2d5a32f303b4f17771d84b72808a406c"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.449682 26484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "42a379d2815b4fe0a91f1b37628819b2"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "a192e0f9ba23d7378bd49bc0f915c03b"
server_key_iv: "2d5a32f303b4f17771d84b72808a406c"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.455705 26484 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260923 18:06:10.459585 26499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:10.461897 26484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:10.462142 26484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "42a379d2815b4fe0a91f1b37628819b2"
format_stamp: "Formatted at 2026-09-23 18:06:10 on dist-test-slave-rvjj"
server_key: "a192e0f9ba23d7378bd49bc0f915c03b"
server_key_iv: "2d5a32f303b4f17771d84b72808a406c"
server_key_version: "encryptionkey@0"
I20260923 18:06:10.462308 26484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:10.475754 26484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:10.481107 26484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:10.481353 26484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:10.483520 26484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:10.483614 26484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:10.483683 26484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:10.483712 26484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:10.497048 26484 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46007
I20260923 18:06:10.497361 26614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46007 every 8 connection(s)
I20260923 18:06:10.498642 26484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:10.501014 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26484
I20260923 18:06:10.501151 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:10.501417 18611 external_mini_cluster.cc:1428] Setting key 8bb8cad39009fd1da1feb1ead33fea11
I20260923 18:06:10.502225 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.484528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:60509 (local address 127.18.44.254:37739)
0923 18:06:10.484726 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.484732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.485734 (+ 1002us) server_negotiation.cc:407] Connection header received
0923 18:06:10.486943 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.486947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.487014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.487085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.488803 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.489839 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.491940 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.492368 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.492565 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.497256 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:10.497261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:10.497265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:10.497311 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:10.499200 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.500055 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.500059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.500061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.500186 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.500642 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.500645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.500647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.500913 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:10.501083 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.501841 (+ 758us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.502034 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:06:10.503583 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.484976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.195:60509)
0923 18:06:10.485535 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.485587 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.486630 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.487526 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.487533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.487925 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:10.488571 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.488591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.490069 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.490072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.491698 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.491715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.492623 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.492628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.492858 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.494177 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:10.494204 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:10.496908 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:10.499427 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.499435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.499456 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.499898 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.500307 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.500310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.500312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.500483 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.501101 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:10.501111 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:10.501484 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.501735 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.502220 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":164,"threads_started":1}
I20260923 18:06:10.505256 26615 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37739
I20260923 18:06:10.505460 26615 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:10.506172 26615 heartbeater.cc:511] Master 127.18.44.254:37739 requested a full tablet report, sending...
I20260923 18:06:10.507766 26131 ts_manager.cc:194] Registered new tserver with Master: 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:10.508646 26131 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:60509
I20260923 18:06:10.516187 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:06:10.517247 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.508121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:38137 (local address 127.18.44.254:37739)
0923 18:06:10.508292 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.508298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.508670 (+ 372us) server_negotiation.cc:407] Connection header received
0923 18:06:10.508858 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.508868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.508936 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.509048 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.509697 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.510253 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.511193 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.511496 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.511611 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.513065 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:10.513068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:10.513070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:10.513097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:10.514852 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.515717 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.515720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.515722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.515817 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.516191 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.516194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.516196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.516441 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:10.516546 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.516925 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.517071 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:06:10.517336 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.508011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.18.44.195:38137)
0923 18:06:10.508514 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.508532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.508693 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.509108 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.509111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.509384 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:10.509534 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.509541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.510400 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.510404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.511053 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.511063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.511655 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.511659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.511763 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.512431 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:10.512444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:10.512880 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:10.515038 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.515043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.515045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.515484 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.515928 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:10.515931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:10.515933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:10.516091 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:10.516558 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:10.516562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:10.516713 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.516845 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.517090 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":166,"threads_started":1}
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:10.550341 26181 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:10.543418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.18.44.254:37739)
0923 18:06:10.543609 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.543613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.543627 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:06:10.543677 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.543680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.543721 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.543792 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.544545 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.545069 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.545934 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.546154 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.546244 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.550181 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60538: BlockingRecv error: recv got EOF from 127.0.0.1:60538 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 18:06:10 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186770, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:10.563166 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.552224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.18.44.254:37739)
0923 18:06:10.552383 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.552388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.552417 (+ 29us) server_negotiation.cc:407] Connection header received
0923 18:06:10.552648 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.552652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.552710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.552816 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:10.553707 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.554341 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.555518 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.555973 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.556120 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.559191 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:10.559195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:10.559198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:10.559239 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:10.561024 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.561672 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.561676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.561677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.561792 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:10.562198 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:10.562200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:10.562201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:10.562438 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:10.562578 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.562854 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.563005 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:06:10.567602 26131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:10.570919 26131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:10.589195 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.582973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.0.0.1:55574)
0923 18:06:10.584038 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.584051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.584149 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.584745 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.584749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.584767 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:10.584924 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.584933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586371 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.586374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.587396 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.587407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.588513 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.588517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.588664 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.588723 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.588846 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.589003 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":3,"thread_start_us":134,"threads_started":1}
I20260923 18:06:10.589417 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.582795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46007 (local address 127.0.0.1:37172)
0923 18:06:10.583594 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.583677 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.583838 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.584652 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.584654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.584669 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:10.584750 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.584756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586127 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.586131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.587673 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.587685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.588813 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.588817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.588957 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.588986 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.589041 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.589147 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":28,"spinlock_wait_cycles":2816,"thread_start_us":142,"threads_started":1}
I20260923 18:06:10.589950 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.583726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.18.44.195:46007)
0923 18:06:10.584179 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.584192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.584216 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:06:10.584288 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.584293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.584486 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.584925 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:10.585065 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586015 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.587835 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.588660 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.589283 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.589572 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.589637 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.589767 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":128,"threads_started":1}
I20260923 18:06:10.590039 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.583349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.18.44.193:45115)
0923 18:06:10.584192 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.584198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.584221 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:06:10.584295 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.584300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.584459 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.584586 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:10.585064 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586192 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.587519 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.588349 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.588491 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.588699 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.589478 (+ 779us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.589821 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":80,"threads_started":1}
I20260923 18:06:10.591166 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.582491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39871 (local address 127.0.0.1:44032)
0923 18:06:10.583593 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.583669 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.583836 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.584546 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.584553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.584576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:10.584748 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.584756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586326 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.586329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.587672 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.587685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.590619 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.590624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.590734 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.590778 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.590885 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.590980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":103,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260923 18:06:10.591384 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.583443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.18.44.194:39871)
0923 18:06:10.583929 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.583944 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.583974 (+ 30us) server_negotiation.cc:407] Connection header received
0923 18:06:10.584068 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.584074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.584332 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.584549 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:10.584890 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.586199 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.588797 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.590408 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.590578 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.590742 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.590994 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.591167 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":106,"threads_started":1}
I20260923 18:06:10.594146 26269 tablet_service.cc:1467] Processing CreateTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 (DEFAULT_TABLE table=test-table [id=818df481b75a4fa981acc7abee6ea2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.594151 26549 tablet_service.cc:1467] Processing CreateTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 (DEFAULT_TABLE table=test-table [id=818df481b75a4fa981acc7abee6ea2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.594740 26407 tablet_service.cc:1467] Processing CreateTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 (DEFAULT_TABLE table=test-table [id=818df481b75a4fa981acc7abee6ea2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.595772 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9c8232a4b34a9a98f7488bcd388938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.595772 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9c8232a4b34a9a98f7488bcd388938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.596093 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9c8232a4b34a9a98f7488bcd388938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.604676 26643 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Bootstrap starting.
I20260923 18:06:10.606482 26645 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Bootstrap starting.
I20260923 18:06:10.606945 26644 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Bootstrap starting.
I20260923 18:06:10.607636 26643 tablet_bootstrap.cc:654] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.608451 26643 log.cc:824] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:10.609653 26645 tablet_bootstrap.cc:654] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.609653 26644 tablet_bootstrap.cc:654] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.610623 26643 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: No bootstrap required, opened a new log
I20260923 18:06:10.610774 26644 log.cc:824] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:10.610775 26645 log.cc:824] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:10.610855 26643 ts_tablet_manager.cc:1397] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:06:10.613288 26644 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: No bootstrap required, opened a new log
I20260923 18:06:10.613468 26645 tablet_bootstrap.cc:492] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: No bootstrap required, opened a new log
I20260923 18:06:10.613567 26644 ts_tablet_manager.cc:1397] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:06:10.613749 26645 ts_tablet_manager.cc:1397] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 18:06:10.614887 26643 raft_consensus.cc:348] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.615211 26643 raft_consensus.cc:374] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.615299 26643 raft_consensus.cc:729] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.615808 26643 consensus_queue.cc:260] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.616811 26475 heartbeater.cc:503] Master 127.18.44.254:37739 was elected leader, sending a full tablet report...
I20260923 18:06:10.617039 26643 ts_tablet_manager.cc:1428] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:06:10.617820 26645 raft_consensus.cc:348] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.618060 26645 raft_consensus.cc:374] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.618124 26645 raft_consensus.cc:729] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.618398 26644 raft_consensus.cc:348] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.618585 26645 consensus_queue.cc:260] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.618724 26644 raft_consensus.cc:374] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.618781 26644 raft_consensus.cc:729] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.619361 26644 consensus_queue.cc:260] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.619990 26645 ts_tablet_manager.cc:1428] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:06:10.620613 26644 ts_tablet_manager.cc:1428] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260923 18:06:10.621120 26335 heartbeater.cc:503] Master 127.18.44.254:37739 was elected leader, sending a full tablet report...
I20260923 18:06:10.621495 26615 heartbeater.cc:503] Master 127.18.44.254:37739 was elected leader, sending a full tablet report...
W20260923 18:06:10.674386 26336 tablet.cc:2369] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:06:10.745265 26476 tablet.cc:2369] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:06:10.750758 26616 tablet.cc:2369] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:06:10.847693 26651 raft_consensus.cc:482] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:10.847956 26651 raft_consensus.cc:504] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.849318 26651 leader_election.cc:288] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:10.854799 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.849662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39871 (local address 127.18.44.193:39589)
0923 18:06:10.850171 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.850190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.850331 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.850925 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.850929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.850967 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:10.851181 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.851189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.852203 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.852207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.853266 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.853278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.854314 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.854317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.854430 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.854460 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.854544 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.854632 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":2,"thread_start_us":139,"threads_started":1}
I20260923 18:06:10.854879 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.849962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:39589 (local address 127.18.44.194:39871)
0923 18:06:10.850152 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.850157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.850290 (+ 133us) server_negotiation.cc:407] Connection header received
0923 18:06:10.850542 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.850546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.850750 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.850851 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:10.851406 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.852057 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.853471 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.854133 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.854298 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.854373 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.854591 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.854701 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 18:06:10.855082 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.849662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46007 (local address 127.18.44.193:32899)
0923 18:06:10.850295 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:10.850314 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:10.850509 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:10.851104 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:10.851107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:10.851128 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:10.851373 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.851384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.852422 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.852426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.853414 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:10.853429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.854549 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.854553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:10.854694 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.854718 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:06:10.854803 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 18:06:10.854893 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":62,"thread_start_us":111,"threads_started":1}
I20260923 18:06:10.855180 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.849982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:32899 (local address 127.18.44.195:46007)
0923 18:06:10.850171 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.850176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.850458 (+ 282us) server_negotiation.cc:407] Connection header received
0923 18:06:10.850840 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.850844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.850910 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.851020 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:10.851599 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.852211 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.853615 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.854296 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.854388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.854441 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.854886 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.855028 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:06:10.856053 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9c8232a4b34a9a98f7488bcd388938" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2" is_pre_election: true
I20260923 18:06:10.856036 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9c8232a4b34a9a98f7488bcd388938" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f" is_pre_election: true
I20260923 18:06:10.856493 26429 raft_consensus.cc:2454] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:10.856493 26569 raft_consensus.cc:2454] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:10.857211 26223 leader_election.cc:302] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2; no voters:
I20260923 18:06:10.857601 26651 raft_consensus.cc:2781] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:10.857723 26651 raft_consensus.cc:482] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:10.857759 26651 raft_consensus.cc:3037] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:10.859697 26651 raft_consensus.cc:504] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.860257 26651 leader_election.cc:288] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Requested vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:10.860826 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9c8232a4b34a9a98f7488bcd388938" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f"
I20260923 18:06:10.860956 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9c8232a4b34a9a98f7488bcd388938" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2"
I20260923 18:06:10.861038 26429 raft_consensus.cc:3037] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:10.861107 26569 raft_consensus.cc:3037] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:10.862869 26429 raft_consensus.cc:2454] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:10.863310 26569 raft_consensus.cc:2454] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:10.863561 26222 leader_election.cc:302] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, b64f6ec176cb43a9853613f95c33a43f; no voters:
I20260923 18:06:10.863848 26651 raft_consensus.cc:2781] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:10.864178 26651 raft_consensus.cc:686] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 LEADER]: Becoming Leader. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Running, Role: LEADER
I20260923 18:06:10.864688 26651 consensus_queue.cc:237] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } }
I20260923 18:06:10.869625 26128 catalog_manager.cc:5495] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } health_report { overall_health: HEALTHY } } }
I20260923 18:06:10.892174 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:10.887437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.18.44.193:45115)
0923 18:06:10.887623 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:10.887627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:10.887646 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:06:10.887829 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:10.887832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:10.887897 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:10.888001 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:10.888346 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.888909 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.890132 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:10.890425 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:10.890524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:10.891061 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:10.891620 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:10.891722 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:10.891919 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 18:06:10.891988 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 18:06:10.900648 26569 raft_consensus.cc:1261] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:10.900694 26429 raft_consensus.cc:1261] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:10.901678 26654 consensus_queue.cc:1035] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:10.902046 26651 consensus_queue.cc:1035] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:10.929173 26132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:10.929400 26132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:10.932791 26132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Sending DeleteTablet for 3 replicas of tablet ae9c8232a4b34a9a98f7488bcd388938
I20260923 18:06:10.933733 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:10 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260923 18:06:10.934454 26666 tablet_replica.cc:330] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: stopping tablet replica
I20260923 18:06:10.934733 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:10 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:37172
I20260923 18:06:10.935205 26666 raft_consensus.cc:2229] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:10.935702 26269 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9c8232a4b34a9a98f7488bcd388938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:10 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260923 18:06:10.935809 26666 raft_consensus.cc:2258] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:10.937310 26132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:10.937816 26132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:10.938167 26666 ts_tablet_manager.cc:1905] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:10.937141 26667 tablet_replica.cc:330] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: stopping tablet replica
I20260923 18:06:10.942091 26666 ts_tablet_manager.cc:1918] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:10.942272 26666 log.cc:1196] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/ae9c8232a4b34a9a98f7488bcd388938
I20260923 18:06:10.942617 26666 ts_tablet_manager.cc:1939] T ae9c8232a4b34a9a98f7488bcd388938 P b64f6ec176cb43a9853613f95c33a43f: Deleting consensus metadata
I20260923 18:06:10.936606 26668 tablet_replica.cc:330] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: stopping tablet replica
I20260923 18:06:10.944717 26118 catalog_manager.cc:4841] TS b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871): tablet ae9c8232a4b34a9a98f7488bcd388938 (table test-table [id=818df481b75a4fa981acc7abee6ea2b3]) successfully deleted
I20260923 18:06:10.944995 26668 raft_consensus.cc:2229] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:10.945797 26668 raft_consensus.cc:2258] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:10.945858 26269 tablet_service.cc:1467] Processing CreateTablet for tablet e616af30d6dc4a8a89d916e5c5559706 (DEFAULT_TABLE table=test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.946240 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616af30d6dc4a8a89d916e5c5559706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.946816 26549 tablet_service.cc:1467] Processing CreateTablet for tablet e616af30d6dc4a8a89d916e5c5559706 (DEFAULT_TABLE table=test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.947678 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616af30d6dc4a8a89d916e5c5559706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.948124 26407 tablet_service.cc:1467] Processing CreateTablet for tablet e616af30d6dc4a8a89d916e5c5559706 (DEFAULT_TABLE table=test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:10.948460 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616af30d6dc4a8a89d916e5c5559706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:10.948601 26668 ts_tablet_manager.cc:1905] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:10.951270 26645 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Bootstrap starting.
I20260923 18:06:10.952193 26667 raft_consensus.cc:2229] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:10.952471 26645 tablet_bootstrap.cc:654] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.952730 26667 raft_consensus.cc:2258] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:10.954021 26643 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Bootstrap starting.
I20260923 18:06:10.954061 26668 ts_tablet_manager.cc:1918] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:10.954208 26668 log.cc:1196] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/ae9c8232a4b34a9a98f7488bcd388938
I20260923 18:06:10.954594 26668 ts_tablet_manager.cc:1939] T ae9c8232a4b34a9a98f7488bcd388938 P 13b87f127ec442fe92d4eb83a4effad1: Deleting consensus metadata
I20260923 18:06:10.955006 26643 tablet_bootstrap.cc:654] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.956403 26116 catalog_manager.cc:4841] TS 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115): tablet ae9c8232a4b34a9a98f7488bcd388938 (table test-table [id=818df481b75a4fa981acc7abee6ea2b3]) successfully deleted
I20260923 18:06:10.956912 26644 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Bootstrap starting.
I20260923 18:06:10.957149 26667 ts_tablet_manager.cc:1905] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:10.957832 26644 tablet_bootstrap.cc:654] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:10.959563 26644 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: No bootstrap required, opened a new log
I20260923 18:06:10.959668 26644 ts_tablet_manager.cc:1397] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 18:06:10.960098 26644 raft_consensus.cc:348] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.960211 26644 raft_consensus.cc:374] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.960240 26644 raft_consensus.cc:729] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.960366 26644 consensus_queue.cc:260] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.960624 26644 ts_tablet_manager.cc:1428] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:06:10.961201 26667 ts_tablet_manager.cc:1918] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:10.961371 26667 log.cc:1196] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/ae9c8232a4b34a9a98f7488bcd388938
I20260923 18:06:10.961995 26667 ts_tablet_manager.cc:1939] T ae9c8232a4b34a9a98f7488bcd388938 P 42a379d2815b4fe0a91f1b37628819b2: Deleting consensus metadata
I20260923 18:06:10.962795 26643 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: No bootstrap required, opened a new log
I20260923 18:06:10.962898 26643 ts_tablet_manager.cc:1397] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260923 18:06:10.963713 26643 raft_consensus.cc:348] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.963852 26643 raft_consensus.cc:374] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.963908 26643 raft_consensus.cc:729] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.964041 26643 consensus_queue.cc:260] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.964078 26119 catalog_manager.cc:4841] TS 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007): tablet ae9c8232a4b34a9a98f7488bcd388938 (table test-table [id=818df481b75a4fa981acc7abee6ea2b3]) successfully deleted
I20260923 18:06:10.964330 26643 ts_tablet_manager.cc:1428] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 18:06:10.965601 26645 tablet_bootstrap.cc:492] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: No bootstrap required, opened a new log
I20260923 18:06:10.965715 26645 ts_tablet_manager.cc:1397] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260923 18:06:10.966189 26645 raft_consensus.cc:348] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.966311 26645 raft_consensus.cc:374] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:10.966341 26645 raft_consensus.cc:729] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Initialized, Role: FOLLOWER
I20260923 18:06:10.966472 26645 consensus_queue.cc:260] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:10.966744 26645 ts_tablet_manager.cc:1428] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:06:11.124809 26649 raft_consensus.cc:482] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:11.124995 26649 raft_consensus.cc:504] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.126295 26649 leader_election.cc:288] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:11.131173 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.126761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:45371 (local address 127.18.44.195:46007)
0923 18:06:11.126980 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.126988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.127228 (+ 240us) server_negotiation.cc:407] Connection header received
0923 18:06:11.127536 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.127538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.127588 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.127651 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:11.128076 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.128608 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.130137 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.130569 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.130695 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.130777 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.130970 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.131044 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:06:11.131206 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.126660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46007 (local address 127.18.44.194:45371)
0923 18:06:11.127071 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.127088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.127401 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.127684 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.127686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.127703 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:11.127939 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.127948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.128725 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.128729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.129964 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.129976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.130721 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.130725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.130837 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.130863 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.130933 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.131023 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":176,"spinlock_wait_cycles":66176,"thread_start_us":114,"threads_started":1}
I20260923 18:06:11.131912 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.126936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:47835 (local address 127.18.44.193:45115)
0923 18:06:11.127073 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.127078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.127311 (+ 233us) server_negotiation.cc:407] Connection header received
0923 18:06:11.127815 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.127818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.127890 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.127964 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:11.128291 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.128939 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.130540 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.131026 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.131168 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.131238 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.131683 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.131799 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:06:11.131915 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.126743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.18.44.194:47835)
0923 18:06:11.127165 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.127180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.127524 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.128008 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.128012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.128030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:11.128182 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.128187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.129112 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.129116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.130315 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.130334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.131168 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.131172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.131488 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.131531 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.131617 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.131714 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":4,"spinlock_wait_cycles":176384,"thread_start_us":219,"threads_started":1}
I20260923 18:06:11.132031 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616af30d6dc4a8a89d916e5c5559706" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2" is_pre_election: true
I20260923 18:06:11.132212 26569 raft_consensus.cc:2454] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64f6ec176cb43a9853613f95c33a43f in term 0.
I20260923 18:06:11.132664 26363 leader_election.cc:302] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a379d2815b4fe0a91f1b37628819b2, b64f6ec176cb43a9853613f95c33a43f; no voters:
I20260923 18:06:11.132733 26289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616af30d6dc4a8a89d916e5c5559706" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b87f127ec442fe92d4eb83a4effad1" is_pre_election: true
I20260923 18:06:11.133004 26649 raft_consensus.cc:2781] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:11.133064 26289 raft_consensus.cc:2454] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64f6ec176cb43a9853613f95c33a43f in term 0.
I20260923 18:06:11.133100 26649 raft_consensus.cc:482] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:11.133193 26649 raft_consensus.cc:3037] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.134588 26649 raft_consensus.cc:504] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.135073 26649 leader_election.cc:288] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 election: Requested vote from peers 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:11.135565 26289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616af30d6dc4a8a89d916e5c5559706" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b87f127ec442fe92d4eb83a4effad1"
I20260923 18:06:11.135722 26289 raft_consensus.cc:3037] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.135774 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616af30d6dc4a8a89d916e5c5559706" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2"
I20260923 18:06:11.135953 26569 raft_consensus.cc:3037] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.136992 26289 raft_consensus.cc:2454] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64f6ec176cb43a9853613f95c33a43f in term 1.
I20260923 18:06:11.137079 26569 raft_consensus.cc:2454] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64f6ec176cb43a9853613f95c33a43f in term 1.
I20260923 18:06:11.137430 26360 leader_election.cc:302] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, b64f6ec176cb43a9853613f95c33a43f; no voters:
I20260923 18:06:11.137790 26649 raft_consensus.cc:2781] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:11.138038 26649 raft_consensus.cc:686] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 1 LEADER]: Becoming Leader. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Running, Role: LEADER
I20260923 18:06:11.138319 26649 consensus_queue.cc:237] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.141165 26132 catalog_manager.cc:5495] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f reported cstate change: term changed from 0 to 1, leader changed from <none> to b64f6ec176cb43a9853613f95c33a43f (127.18.44.194). New cstate: current_term: 1 leader_uuid: "b64f6ec176cb43a9853613f95c33a43f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } health_report { overall_health: UNKNOWN } } }
Sep 23 18:06:11 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186769, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.254@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:11.186594 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.173850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37739 (local address 127.0.0.1:60560)
0923 18:06:11.174256 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.174273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.174411 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.174730 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.174732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.175484 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:11.175588 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.175595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.176756 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.176759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.177381 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.177391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.178078 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.178082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.178196 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.178951 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:11.178964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:11.180979 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:11.184093 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:11.184110 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:11.184118 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:11.184535 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:11.185124 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:11.185128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:11.185130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:11.185283 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:11.185940 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:11.185947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:11.186108 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.186247 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.186429 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":158,"threads_started":1}
I20260923 18:06:11.186882 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.174020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.18.44.254:37739)
0923 18:06:11.174486 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.174498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.174515 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:06:11.174555 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.174557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.174596 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.174696 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:11.175811 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.176568 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.177581 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.177863 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.178046 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.181464 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:11.181469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:11.181472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:11.181516 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:11.183840 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:11.184786 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:11.184791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:11.184793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:11.184950 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:11.185420 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:11.185423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:11.185425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:11.185785 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:11.185899 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.186454 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.186654 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":172,"threads_started":1}
I20260923 18:06:11.192734 26132 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:60560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:06:11.193946 26132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:11.201874 26269 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab2b12d2fb46bfbf9688bf26b48245 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7160a95d75924b08813490e8a2e19319]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:06:11.201900 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab2b12d2fb46bfbf9688bf26b48245 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7160a95d75924b08813490e8a2e19319]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:06:11.201900 26549 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab2b12d2fb46bfbf9688bf26b48245 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7160a95d75924b08813490e8a2e19319]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:06:11.202445 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab2b12d2fb46bfbf9688bf26b48245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.202468 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab2b12d2fb46bfbf9688bf26b48245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.202463 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab2b12d2fb46bfbf9688bf26b48245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.206210 26645 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: Bootstrap starting.
I20260923 18:06:11.206223 26643 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: Bootstrap starting.
I20260923 18:06:11.206241 26644 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: Bootstrap starting.
I20260923 18:06:11.207278 26643 tablet_bootstrap.cc:654] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.207278 26645 tablet_bootstrap.cc:654] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.207301 26644 tablet_bootstrap.cc:654] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.208547 26645 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: No bootstrap required, opened a new log
I20260923 18:06:11.208588 26644 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: No bootstrap required, opened a new log
I20260923 18:06:11.208654 26645 ts_tablet_manager.cc:1397] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:06:11.208671 26644 ts_tablet_manager.cc:1397] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 18:06:11.208679 26643 tablet_bootstrap.cc:492] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: No bootstrap required, opened a new log
I20260923 18:06:11.208748 26643 ts_tablet_manager.cc:1397] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:06:11.209090 26644 raft_consensus.cc:348] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209122 26645 raft_consensus.cc:348] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209213 26644 raft_consensus.cc:374] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.209160 26643 raft_consensus.cc:348] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209237 26645 raft_consensus.cc:374] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.209244 26644 raft_consensus.cc:729] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.209275 26643 raft_consensus.cc:374] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.209281 26645 raft_consensus.cc:729] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.209308 26643 raft_consensus.cc:729] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.209375 26644 consensus_queue.cc:260] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209407 26645 consensus_queue.cc:260] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209415 26643 consensus_queue.cc:260] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.209635 26644 ts_tablet_manager.cc:1428] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:06:11.209663 26645 ts_tablet_manager.cc:1428] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:11.209663 26643 ts_tablet_manager.cc:1428] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:11.209852 26654 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.209848 26650 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.209981 26650 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.209981 26654 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.210624 26649 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.210776 26649 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.490028 26654 raft_consensus.cc:482] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:11.490187 26654 raft_consensus.cc:504] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.490614 26654 leader_election.cc:288] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:11.491289 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab2b12d2fb46bfbf9688bf26b48245" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f" is_pre_election: true
I20260923 18:06:11.491369 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab2b12d2fb46bfbf9688bf26b48245" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2" is_pre_election: true
I20260923 18:06:11.491483 26429 raft_consensus.cc:2454] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:11.491494 26569 raft_consensus.cc:2454] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:11.491838 26223 leader_election.cc:302] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2; no voters:
I20260923 18:06:11.492020 26654 raft_consensus.cc:2781] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:11.492095 26654 raft_consensus.cc:482] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:11.492125 26654 raft_consensus.cc:3037] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.493379 26654 raft_consensus.cc:504] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.493825 26654 leader_election.cc:288] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Requested vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:11.494343 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab2b12d2fb46bfbf9688bf26b48245" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f"
I20260923 18:06:11.494520 26429 raft_consensus.cc:3037] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.494514 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab2b12d2fb46bfbf9688bf26b48245" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2"
I20260923 18:06:11.494657 26569 raft_consensus.cc:3037] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.495608 26429 raft_consensus.cc:2454] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:11.495661 26569 raft_consensus.cc:2454] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:11.495975 26222 leader_election.cc:302] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, b64f6ec176cb43a9853613f95c33a43f; no voters:
I20260923 18:06:11.496186 26654 raft_consensus.cc:2781] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:11.496383 26654 raft_consensus.cc:686] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 LEADER]: Becoming Leader. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Running, Role: LEADER
I20260923 18:06:11.496537 26654 consensus_queue.cc:237] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.497217 26686 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: TxnStatusTablet state changed. Reason: New leader 13b87f127ec442fe92d4eb83a4effad1. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.497326 26686 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:06:11.498452 26131 catalog_manager.cc:5495] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } health_report { overall_health: UNKNOWN } } }
I20260923 18:06:11.500960 26688 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:06:11.501176 26688 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:06:11.576897 26649 consensus_queue.cc:1035] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.579828 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.575846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.0.0.1:44228)
0923 18:06:11.576113 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.576129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.576253 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.576537 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.576540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.576547 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:06:11.576673 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.576679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.577525 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.577529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.578376 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.578389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.578980 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.578984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.579076 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.579150 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.579512 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.579526 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.579580 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.579681 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260923 18:06:11.580247 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.575911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.18.44.193:45115)
0923 18:06:11.576068 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.576073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.576194 (+ 121us) server_negotiation.cc:407] Connection header received
0923 18:06:11.576372 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.576375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.576425 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.576507 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:11.576774 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.577402 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.578504 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.578842 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.578948 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.579325 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.579410 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.579476 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.580027 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.580088 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:06:11.581733 26674 consensus_queue.cc:1035] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.585048 26429 raft_consensus.cc:1261] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:11.585507 26686 consensus_queue.cc:1035] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.586015 26569 raft_consensus.cc:1261] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:11.586455 26686 consensus_queue.cc:1035] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.587180 26674 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: TxnStatusTablet state changed. Reason: New leader 13b87f127ec442fe92d4eb83a4effad1. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.587711 26674 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.588935 26697 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: TxnStatusTablet state changed. Reason: New leader 13b87f127ec442fe92d4eb83a4effad1. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.589084 26697 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.591118 26686 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.591711 26696 mvcc.cc:205] Tried to move back new op lower bound from 7332605016403451904 to 7332605016051421184. Current Snapshot: MvccSnapshot[applied={T|T < 7332605016403451904}]
I20260923 18:06:11.592212 26697 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.592335 26697 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 42a379d2815b4fe0a91f1b37628819b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.592762 26686 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:06:11.595042 26692 mvcc.cc:205] Tried to move back new op lower bound from 7332605016403451904 to 7332605016051421184. Current Snapshot: MvccSnapshot[applied={T|T < 7332605016403451904}]
I20260923 18:06:11.592978 26693 mvcc.cc:205] Tried to move back new op lower bound from 7332605016403451904 to 7332605016051421184. Current Snapshot: MvccSnapshot[applied={T|T < 7332605016403451904 or (T in {7332605016403451904})}]
I20260923 18:06:11.595749 26686 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.595783 26674 tablet_replica.cc:439] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } } }
I20260923 18:06:11.595877 26686 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P 13b87f127ec442fe92d4eb83a4effad1: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:06:11.595902 26674 tablet_replica.cc:442] T 4eab2b12d2fb46bfbf9688bf26b48245 P b64f6ec176cb43a9853613f95c33a43f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:06:11.602700 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.598395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.18.44.194:39871)
0923 18:06:11.598717 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.598722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.598757 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:06:11.598851 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.598854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.598919 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.599018 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:11.599316 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.599861 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.600767 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.601016 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.601106 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.601473 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.602104 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.602257 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.602442 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.602517 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260923 18:06:11.616006 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.611761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.18.44.194:52901)
0923 18:06:11.612150 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.612165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.612277 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.612594 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.612597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.612603 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:06:11.612720 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.612725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.613547 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.613551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.614544 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.614557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.615108 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.615112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.615199 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.615303 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.615636 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.615643 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.615715 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.615813 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
I20260923 18:06:11.616070 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.611915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:52901 (local address 127.18.44.193:45115)
0923 18:06:11.612204 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.612208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.612230 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:06:11.612447 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.612450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.612499 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.612568 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:11.612876 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.613397 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.614729 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.614979 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.615073 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.615463 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.615551 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.615609 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.615894 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.615959 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260923 18:06:11.653173 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.647876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:50501 (local address 127.18.44.194:39871)
0923 18:06:11.648024 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.648029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.648175 (+ 146us) server_negotiation.cc:407] Connection header received
0923 18:06:11.648300 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.648303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.648365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.648457 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:11.648818 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.650005 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.651343 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.651927 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.652056 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.652580 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.652685 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.652762 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.652942 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.653011 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:06:11.653278 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.647437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39871 (local address 127.18.44.193:50501)
0923 18:06:11.648083 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.648099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.648211 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.648498 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.648500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.648507 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:06:11.648610 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.648615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.650154 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.650158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.651030 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.651040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.652216 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.652221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.652345 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.652417 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.652813 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.652823 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.652926 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.653021 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":146,"threads_started":1}
I20260923 18:06:11.692240 26131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:11.692432 26131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:11.695389 26131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Sending DeleteTablet for 3 replicas of tablet e616af30d6dc4a8a89d916e5c5559706
I20260923 18:06:11.696381 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet e616af30d6dc4a8a89d916e5c5559706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260923 18:06:11.696424 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet e616af30d6dc4a8a89d916e5c5559706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:37172
I20260923 18:06:11.696691 26269 tablet_service.cc:1514] Processing DeleteTablet for tablet e616af30d6dc4a8a89d916e5c5559706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260923 18:06:11.697045 26718 tablet_replica.cc:330] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: stopping tablet replica
I20260923 18:06:11.697099 26717 tablet_replica.cc:330] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: stopping tablet replica
I20260923 18:06:11.697332 26717 raft_consensus.cc:2229] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:11.697484 26717 raft_consensus.cc:2258] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.697736 26719 tablet_replica.cc:330] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: stopping tablet replica
I20260923 18:06:11.697870 26719 raft_consensus.cc:2229] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:11.697990 26719 raft_consensus.cc:2258] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.698971 26717 ts_tablet_manager.cc:1905] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.697371 26718 raft_consensus.cc:2229] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:11.699826 26719 ts_tablet_manager.cc:1905] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.700497 26718 raft_consensus.cc:2258] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.703230 26717 ts_tablet_manager.cc:1918] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:06:11.703392 26717 log.cc:1196] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/e616af30d6dc4a8a89d916e5c5559706
I20260923 18:06:11.703646 26717 ts_tablet_manager.cc:1939] T e616af30d6dc4a8a89d916e5c5559706 P 42a379d2815b4fe0a91f1b37628819b2: Deleting consensus metadata
I20260923 18:06:11.704226 26719 ts_tablet_manager.cc:1918] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:06:11.704316 26719 log.cc:1196] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/e616af30d6dc4a8a89d916e5c5559706
I20260923 18:06:11.704746 26719 ts_tablet_manager.cc:1939] T e616af30d6dc4a8a89d916e5c5559706 P 13b87f127ec442fe92d4eb83a4effad1: Deleting consensus metadata
I20260923 18:06:11.705302 26718 ts_tablet_manager.cc:1905] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.705641 26116 catalog_manager.cc:4841] TS 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115): tablet e616af30d6dc4a8a89d916e5c5559706 (table test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]) successfully deleted
I20260923 18:06:11.706616 26119 catalog_manager.cc:4841] TS 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007): tablet e616af30d6dc4a8a89d916e5c5559706 (table test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]) successfully deleted
I20260923 18:06:11.709538 26718 ts_tablet_manager.cc:1918] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:06:11.709753 26718 log.cc:1196] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/e616af30d6dc4a8a89d916e5c5559706
I20260923 18:06:11.710322 26718 ts_tablet_manager.cc:1939] T e616af30d6dc4a8a89d916e5c5559706 P b64f6ec176cb43a9853613f95c33a43f: Deleting consensus metadata
I20260923 18:06:11.711436 26118 catalog_manager.cc:4841] TS b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871): tablet e616af30d6dc4a8a89d916e5c5559706 (table test-table [id=49ead70fd3b946ffbf7d8bfe7dd3f7e6]) successfully deleted
Sep 23 18:06:11 dist-test-slave-rvjj krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:06:11 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:11 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:11.733127 26727 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:11.725374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.18.44.254:37739)
0923 18:06:11.725663 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.725668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.725686 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:06:11.725737 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.725740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.725791 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.725892 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:11.726767 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.728020 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.729043 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.729425 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.729546 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.732957 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44566: BlockingRecv error: recv got EOF from 127.0.0.1:44566 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
Sep 23 18:06:11 dist-test-slave-rvjj krb5kdc[26087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM
I20260923 18:06:11.744998 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.734749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.18.44.254:37739)
0923 18:06:11.734896 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.734901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.735113 (+ 212us) server_negotiation.cc:407] Connection header received
0923 18:06:11.735286 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.735291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.735357 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.735433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:11.736216 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.736742 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.737880 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.738108 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.738203 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.741368 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:11.741371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:11.741374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:11.741405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:11.742963 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:11.743611 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:11.743615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:11.743617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:11.743718 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:11.744017 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:11.744019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:11.744020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:11.744299 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:11.744415 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.744709 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.744836 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:06:11.748059 26131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:11.748525 26131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:11.754484 26269 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be (DEFAULT_TABLE table=test-table [id=72ede8e91fac4d818db13f07dcb6eec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.754884 26407 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be (DEFAULT_TABLE table=test-table [id=72ede8e91fac4d818db13f07dcb6eec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.754971 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8539a4737a4133b30eb2b3536dc3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.754895 26549 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be (DEFAULT_TABLE table=test-table [id=72ede8e91fac4d818db13f07dcb6eec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.755169 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8539a4737a4133b30eb2b3536dc3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.755200 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8539a4737a4133b30eb2b3536dc3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.758141 26733 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Bootstrap starting.
I20260923 18:06:11.759317 26733 tablet_bootstrap.cc:654] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.759508 26734 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Bootstrap starting.
I20260923 18:06:11.760528 26734 tablet_bootstrap.cc:654] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.760627 26733 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: No bootstrap required, opened a new log
I20260923 18:06:11.760725 26733 ts_tablet_manager.cc:1397] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 18:06:11.761219 26735 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Bootstrap starting.
I20260923 18:06:11.761214 26733 raft_consensus.cc:348] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.761344 26733 raft_consensus.cc:374] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.761374 26733 raft_consensus.cc:729] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.761540 26733 consensus_queue.cc:260] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.762614 26735 tablet_bootstrap.cc:654] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.762698 26733 ts_tablet_manager.cc:1428] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 18:06:11.763749 26710 raft_consensus.cc:482] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:11.763891 26710 raft_consensus.cc:504] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.764410 26710 leader_election.cc:288] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:11.764899 26289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b87f127ec442fe92d4eb83a4effad1" is_pre_election: true
I20260923 18:06:11.765012 26735 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: No bootstrap required, opened a new log
I20260923 18:06:11.765120 26735 ts_tablet_manager.cc:1397] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:11.765641 26735 raft_consensus.cc:348] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.765775 26735 raft_consensus.cc:374] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.765808 26735 raft_consensus.cc:729] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.765928 26735 consensus_queue.cc:260] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
W20260923 18:06:11.766054 26360 leader_election.cc:341] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115): Illegal state: must be running to vote when last-logged opid is not known
I20260923 18:06:11.766184 26735 ts_tablet_manager.cc:1428] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:06:11.766265 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "b64f6ec176cb43a9853613f95c33a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2" is_pre_election: true
W20260923 18:06:11.766752 26363 leader_election.cc:341] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007): Illegal state: must be running to vote when last-logged opid is not known
I20260923 18:06:11.766839 26363 leader_election.cc:302] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b64f6ec176cb43a9853613f95c33a43f; no voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2
I20260923 18:06:11.767100 26710 raft_consensus.cc:2726] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 18:06:11.767128 26734 tablet_bootstrap.cc:492] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: No bootstrap required, opened a new log
I20260923 18:06:11.767222 26734 ts_tablet_manager.cc:1397] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260923 18:06:11.767726 26734 raft_consensus.cc:348] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.767861 26734 raft_consensus.cc:374] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.767891 26734 raft_consensus.cc:729] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.768014 26734 consensus_queue.cc:260] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.768895 26734 ts_tablet_manager.cc:1428] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 18:06:11.887964 26697 raft_consensus.cc:482] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:11.888230 26697 raft_consensus.cc:504] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.889464 26697 leader_election.cc:288] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115), b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871)
I20260923 18:06:11.893640 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.889796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:37925 (local address 127.18.44.193:45115)
0923 18:06:11.890034 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.890040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.890214 (+ 174us) server_negotiation.cc:407] Connection header received
0923 18:06:11.890450 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.890453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.890601 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.890700 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:11.891025 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.891612 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.892673 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.893086 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.893203 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.893260 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.893434 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.893509 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260923 18:06:11.893669 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.889728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.18.44.195:37925)
0923 18:06:11.890114 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.890131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.890308 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.890729 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.890732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.890749 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:11.890916 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.890923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.891755 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.891757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.892535 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.892543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.893212 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.893215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.893319 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.893340 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.893407 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.893485 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":28,"spinlock_wait_cycles":22656,"thread_start_us":79,"threads_started":1}
I20260923 18:06:11.894006 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.890333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:56525 (local address 127.18.44.194:39871)
0923 18:06:11.890445 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.890449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.890463 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:06:11.890518 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.890521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.890566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.890654 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:11.890951 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.891517 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.892837 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.893399 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.893524 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.893576 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.893792 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.893893 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 18:06:11.894182 26289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "42a379d2815b4fe0a91f1b37628819b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b87f127ec442fe92d4eb83a4effad1" is_pre_election: true
I20260923 18:06:11.894322 26289 raft_consensus.cc:2454] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a379d2815b4fe0a91f1b37628819b2 in term 0.
I20260923 18:06:11.894484 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.889852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39871 (local address 127.18.44.195:56525)
0923 18:06:11.890176 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:11.890189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:11.890308 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:11.890670 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:11.890674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:11.890692 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:11.890841 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.890849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.891619 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.891622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.892529 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:11.892541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.893531 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.893535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:11.893655 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.893674 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:06:11.894193 (+ 519us) client_negotiation.cc:240] Negotiation successful
0923 18:06:11.894337 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":3,"spinlock_wait_cycles":7168,"thread_start_us":110,"threads_started":1}
I20260923 18:06:11.894681 26500 leader_election.cc:302] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2; no voters:
I20260923 18:06:11.894914 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "42a379d2815b4fe0a91f1b37628819b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f" is_pre_election: true
I20260923 18:06:11.894972 26697 raft_consensus.cc:2781] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:11.895020 26697 raft_consensus.cc:482] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:11.895028 26429 raft_consensus.cc:2454] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a379d2815b4fe0a91f1b37628819b2 in term 0.
I20260923 18:06:11.895046 26697 raft_consensus.cc:3037] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.896092 26697 raft_consensus.cc:504] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.896447 26697 leader_election.cc:288] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [CANDIDATE]: Term 1 election: Requested vote from peers 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115), b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871)
I20260923 18:06:11.896790 26289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "42a379d2815b4fe0a91f1b37628819b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b87f127ec442fe92d4eb83a4effad1"
I20260923 18:06:11.896917 26289 raft_consensus.cc:3037] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.896906 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8539a4737a4133b30eb2b3536dc3be" candidate_uuid: "42a379d2815b4fe0a91f1b37628819b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f"
I20260923 18:06:11.897069 26429 raft_consensus.cc:3037] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:11.897888 26289 raft_consensus.cc:2454] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a379d2815b4fe0a91f1b37628819b2 in term 1.
I20260923 18:06:11.897971 26429 raft_consensus.cc:2454] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a379d2815b4fe0a91f1b37628819b2 in term 1.
I20260923 18:06:11.898165 26500 leader_election.cc:302] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2; no voters:
I20260923 18:06:11.898339 26697 raft_consensus.cc:2781] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:11.898442 26697 raft_consensus.cc:686] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 1 LEADER]: Becoming Leader. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Running, Role: LEADER
I20260923 18:06:11.898684 26697 consensus_queue.cc:237] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.901484 26131 catalog_manager.cc:5495] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "42a379d2815b4fe0a91f1b37628819b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } health_report { overall_health: HEALTHY } } }
I20260923 18:06:11.930228 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:11.925650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.18.44.195:46007)
0923 18:06:11.926025 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:11.926040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:11.926052 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:06:11.926132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:11.926134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:11.926188 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:11.926308 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:11.926582 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.927439 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.928422 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:11.928669 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:11.928864 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:11.929080 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:11.929739 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:11.929884 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:11.929987 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 18:06:11.930053 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260923 18:06:11.933344 26429 raft_consensus.cc:1261] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Refusing update from remote peer 42a379d2815b4fe0a91f1b37628819b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:11.933372 26289 raft_consensus.cc:1261] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Refusing update from remote peer 42a379d2815b4fe0a91f1b37628819b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:11.933943 26650 consensus_queue.cc:1035] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.934139 26697 consensus_queue.cc:1035] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:11.949939 26128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:11.950115 26128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:11.952752 26128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Sending DeleteTablet for 3 replicas of tablet 4f8539a4737a4133b30eb2b3536dc3be
I20260923 18:06:11.953544 26269 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260923 18:06:11.953600 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:37172
I20260923 18:06:11.953799 26719 tablet_replica.cc:330] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: stopping tablet replica
I20260923 18:06:11.953789 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8539a4737a4133b30eb2b3536dc3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:11 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260923 18:06:11.953920 26719 raft_consensus.cc:2229] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:11.953962 26718 tablet_replica.cc:330] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: stopping tablet replica
I20260923 18:06:11.954051 26718 raft_consensus.cc:2229] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:11.954071 26719 raft_consensus.cc:2258] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.954195 26718 raft_consensus.cc:2258] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.954241 26717 tablet_replica.cc:330] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: stopping tablet replica
I20260923 18:06:11.954348 26717 raft_consensus.cc:2229] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:11.954675 26717 raft_consensus.cc:2258] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:11.955690 26719 ts_tablet_manager.cc:1905] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.955687 26718 ts_tablet_manager.cc:1905] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.958405 26719 ts_tablet_manager.cc:1918] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:11.958510 26719 log.cc:1196] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/4f8539a4737a4133b30eb2b3536dc3be
I20260923 18:06:11.958474 26718 ts_tablet_manager.cc:1918] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:11.958792 26718 log.cc:1196] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/4f8539a4737a4133b30eb2b3536dc3be
I20260923 18:06:11.958804 26719 ts_tablet_manager.cc:1939] T 4f8539a4737a4133b30eb2b3536dc3be P 13b87f127ec442fe92d4eb83a4effad1: Deleting consensus metadata
I20260923 18:06:11.959180 26718 ts_tablet_manager.cc:1939] T 4f8539a4737a4133b30eb2b3536dc3be P b64f6ec176cb43a9853613f95c33a43f: Deleting consensus metadata
I20260923 18:06:11.959417 26717 ts_tablet_manager.cc:1905] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:11.961623 26717 ts_tablet_manager.cc:1918] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:11.961709 26717 log.cc:1196] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/4f8539a4737a4133b30eb2b3536dc3be
I20260923 18:06:11.961911 26717 ts_tablet_manager.cc:1939] T 4f8539a4737a4133b30eb2b3536dc3be P 42a379d2815b4fe0a91f1b37628819b2: Deleting consensus metadata
I20260923 18:06:11.962240 26118 catalog_manager.cc:4841] TS b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871): tablet 4f8539a4737a4133b30eb2b3536dc3be (table test-table [id=72ede8e91fac4d818db13f07dcb6eec6]) successfully deleted
I20260923 18:06:11.962258 26116 catalog_manager.cc:4841] TS 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115): tablet 4f8539a4737a4133b30eb2b3536dc3be (table test-table [id=72ede8e91fac4d818db13f07dcb6eec6]) successfully deleted
I20260923 18:06:11.962927 26128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:06:11.963217 26119 catalog_manager.cc:4841] TS 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007): tablet 4f8539a4737a4133b30eb2b3536dc3be (table test-table [id=72ede8e91fac4d818db13f07dcb6eec6]) successfully deleted
W20260923 18:06:11.963481 26128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:11.968624 26269 tablet_service.cc:1467] Processing CreateTablet for tablet f3b8c62457174576b574b9523639dc20 (DEFAULT_TABLE table=test-table [id=7af01c0331d141c68ae1627bd6b38453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.968883 26549 tablet_service.cc:1467] Processing CreateTablet for tablet f3b8c62457174576b574b9523639dc20 (DEFAULT_TABLE table=test-table [id=7af01c0331d141c68ae1627bd6b38453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.968972 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b8c62457174576b574b9523639dc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.969144 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b8c62457174576b574b9523639dc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.971498 26735 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: Bootstrap starting.
I20260923 18:06:11.971942 26407 tablet_service.cc:1467] Processing CreateTablet for tablet f3b8c62457174576b574b9523639dc20 (DEFAULT_TABLE table=test-table [id=7af01c0331d141c68ae1627bd6b38453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:11.972223 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b8c62457174576b574b9523639dc20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:11.972396 26735 tablet_bootstrap.cc:654] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.972963 26734 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Bootstrap starting.
I20260923 18:06:11.973446 26735 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: No bootstrap required, opened a new log
I20260923 18:06:11.973515 26735 ts_tablet_manager.cc:1397] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:06:11.973853 26735 raft_consensus.cc:348] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.973920 26734 tablet_bootstrap.cc:654] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.973940 26735 raft_consensus.cc:374] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.973960 26735 raft_consensus.cc:729] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.974068 26735 consensus_queue.cc:260] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.974292 26735 ts_tablet_manager.cc:1428] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:06:11.974736 26733 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Bootstrap starting.
I20260923 18:06:11.975816 26733 tablet_bootstrap.cc:654] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:11.977249 26734 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: No bootstrap required, opened a new log
I20260923 18:06:11.977370 26734 ts_tablet_manager.cc:1397] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260923 18:06:11.977893 26734 raft_consensus.cc:348] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.978030 26734 raft_consensus.cc:374] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.978053 26734 raft_consensus.cc:729] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a379d2815b4fe0a91f1b37628819b2, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.978183 26734 consensus_queue.cc:260] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.978441 26734 ts_tablet_manager.cc:1428] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:06:11.984239 26733 tablet_bootstrap.cc:492] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: No bootstrap required, opened a new log
I20260923 18:06:11.984375 26733 ts_tablet_manager.cc:1397] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260923 18:06:11.984866 26733 raft_consensus.cc:348] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.985009 26733 raft_consensus.cc:374] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:11.985052 26733 raft_consensus.cc:729] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f6ec176cb43a9853613f95c33a43f, State: Initialized, Role: FOLLOWER
I20260923 18:06:11.985191 26733 consensus_queue.cc:260] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:11.985491 26733 ts_tablet_manager.cc:1428] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:06:12.130821 26654 raft_consensus.cc:482] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:12.131021 26654 raft_consensus.cc:504] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:12.131753 26654 leader_election.cc:288] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:12.132266 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b8c62457174576b574b9523639dc20" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2" is_pre_election: true
I20260923 18:06:12.132346 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b8c62457174576b574b9523639dc20" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f" is_pre_election: true
I20260923 18:06:12.132424 26569 raft_consensus.cc:2454] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:12.132493 26429 raft_consensus.cc:2454] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 0.
I20260923 18:06:12.132807 26223 leader_election.cc:302] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2; no voters:
I20260923 18:06:12.133008 26654 raft_consensus.cc:2781] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:12.133095 26654 raft_consensus.cc:482] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:12.133147 26654 raft_consensus.cc:3037] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:12.134454 26654 raft_consensus.cc:504] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:12.134883 26654 leader_election.cc:288] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Requested vote from peers b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871), 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007)
I20260923 18:06:12.135608 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b8c62457174576b574b9523639dc20" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a379d2815b4fe0a91f1b37628819b2"
I20260923 18:06:12.135607 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b8c62457174576b574b9523639dc20" candidate_uuid: "13b87f127ec442fe92d4eb83a4effad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64f6ec176cb43a9853613f95c33a43f"
I20260923 18:06:12.135807 26569 raft_consensus.cc:3037] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:12.135835 26429 raft_consensus.cc:3037] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:12.137357 26569 raft_consensus.cc:2454] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:12.137354 26429 raft_consensus.cc:2454] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b87f127ec442fe92d4eb83a4effad1 in term 1.
I20260923 18:06:12.137848 26222 leader_election.cc:302] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13b87f127ec442fe92d4eb83a4effad1, 42a379d2815b4fe0a91f1b37628819b2, b64f6ec176cb43a9853613f95c33a43f; no voters:
I20260923 18:06:12.138062 26654 raft_consensus.cc:2781] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:12.138258 26654 raft_consensus.cc:686] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 LEADER]: Becoming Leader. State: Replica: 13b87f127ec442fe92d4eb83a4effad1, State: Running, Role: LEADER
I20260923 18:06:12.138520 26654 consensus_queue.cc:237] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } }
I20260923 18:06:12.140484 26128 catalog_manager.cc:5495] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "13b87f127ec442fe92d4eb83a4effad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b87f127ec442fe92d4eb83a4effad1" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 } health_report { overall_health: UNKNOWN } } }
I20260923 18:06:12.184535 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:12.181023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.18.44.193:45115)
0923 18:06:12.181213 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:12.181217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:12.181233 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:12.181287 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:12.181289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:12.181336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:12.181422 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:12.181776 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.182313 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.183214 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.183530 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.183631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:12.183967 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:12.184051 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:12.184125 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:12.184311 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 18:06:12.184365 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 18:06:12.194746 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:12.191609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:36133 (local address 127.18.44.193:45115)
0923 18:06:12.191797 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:12.191802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:12.191878 (+ 76us) server_negotiation.cc:407] Connection header received
0923 18:06:12.192005 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:12.192008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:12.192049 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:12.192107 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:12.192406 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.192906 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.193709 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.193920 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.194002 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:12.194239 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:12.194313 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:12.194370 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:12.194479 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 18:06:12.194575 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":43}
I20260923 18:06:12.194746 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:12.191470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.18.44.193:36133)
0923 18:06:12.191797 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:12.191813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:12.191921 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:12.192120 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:12.192123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:12.192129 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:06:12.192314 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:12.192321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.192999 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.193002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:12.193595 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:12.193603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:12.194016 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:12.194020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:12.194120 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:12.194144 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:06:12.194381 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:06:12.194409 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:06:12.194470 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 18:06:12.194601 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":59,"thread_start_us":179,"threads_started":1}
I20260923 18:06:12.200978 26569 raft_consensus.cc:1261] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:12.200945 26429 raft_consensus.cc:1261] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Refusing update from remote peer 13b87f127ec442fe92d4eb83a4effad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:12.201412 26752 consensus_queue.cc:1035] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64f6ec176cb43a9853613f95c33a43f" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:12.201802 26712 consensus_queue.cc:1035] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a379d2815b4fe0a91f1b37628819b2" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:12.239626 26132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:12.239909 26132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:12.242468 26132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1855a2c327a246efac22606e31b55653: Sending DeleteTablet for 3 replicas of tablet f3b8c62457174576b574b9523639dc20
I20260923 18:06:12.243302 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b8c62457174576b574b9523639dc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:12 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:37172
I20260923 18:06:12.243386 26268 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b8c62457174576b574b9523639dc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:12 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260923 18:06:12.243585 26717 tablet_replica.cc:330] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: stopping tablet replica
I20260923 18:06:12.243541 26407 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b8c62457174576b574b9523639dc20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:12 UTC) from {username='oryx', principal='oryx/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260923 18:06:12.243610 26719 tablet_replica.cc:330] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1: stopping tablet replica
I20260923 18:06:12.243721 26719 raft_consensus.cc:2229] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:12.243742 26717 raft_consensus.cc:2229] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:12.243858 26717 raft_consensus.cc:2258] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:12.243937 26719 raft_consensus.cc:2258] T f3b8c62457174576b574b9523639dc20 P 13b87f127ec442fe92d4eb83a4effad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:12.244028 26718 tablet_replica.cc:330] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: stopping tablet replica
I20260923 18:06:12.244163 26718 raft_consensus.cc:2229] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:12.244264 26718 raft_consensus.cc:2258] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:12.244879 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26204
I20260923 18:06:12.245188 26717 ts_tablet_manager.cc:1905] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:12.249150 26717 ts_tablet_manager.cc:1918] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:06:12.249269 26717 log.cc:1196] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/f3b8c62457174576b574b9523639dc20
I20260923 18:06:12.249506 26717 ts_tablet_manager.cc:1939] T f3b8c62457174576b574b9523639dc20 P 42a379d2815b4fe0a91f1b37628819b2: Deleting consensus metadata
I20260923 18:06:12.250890 26119 catalog_manager.cc:4841] TS 42a379d2815b4fe0a91f1b37628819b2 (127.18.44.195:46007): tablet f3b8c62457174576b574b9523639dc20 (table test-table [id=7af01c0331d141c68ae1627bd6b38453]) successfully deleted
I20260923 18:06:12.252038 26718 ts_tablet_manager.cc:1905] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:12.254638 26718 ts_tablet_manager.cc:1918] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:06:12.254720 26718 log.cc:1196] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/f3b8c62457174576b574b9523639dc20
I20260923 18:06:12.254971 26718 ts_tablet_manager.cc:1939] T f3b8c62457174576b574b9523639dc20 P b64f6ec176cb43a9853613f95c33a43f: Deleting consensus metadata
I20260923 18:06:12.255873 26118 catalog_manager.cc:4841] TS b64f6ec176cb43a9853613f95c33a43f (127.18.44.194:39871): tablet f3b8c62457174576b574b9523639dc20 (table test-table [id=7af01c0331d141c68ae1627bd6b38453]) successfully deleted
W20260923 18:06:12.256704 26116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:45115 (error 108)
I20260923 18:06:12.257035 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26344
I20260923 18:06:12.259208 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:12.258414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45115 (local address 127.0.0.1:44248)
0923 18:06:12.259017 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:12.259091 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":61,"threads_started":1}
W20260923 18:06:12.259394 26116 catalog_manager.cc:4584] TS 13b87f127ec442fe92d4eb83a4effad1 (127.18.44.193:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3b8c62457174576b574b9523639dc20: Network error: Client connection negotiation failed: client connection to 127.18.44.193:45115: connect: Connection refused (error 111)
I20260923 18:06:12.263818 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26484
I20260923 18:06:12.270457 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26100
2026-09-23T18:06:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4191 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:12 dist-test-slave-rvjj krb5kdc[26763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:12 dist-test-slave-rvjj krb5kdc[26763](info): set up 2 sockets
Sep 23 18:06:12 dist-test-slave-rvjj krb5kdc[26763](info): commencing operation
krb5kdc: starting...
W20260923 18:06:13.302798 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:13 dist-test-slave-rvjj krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:06:13.489382 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:35927
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:13.659979 26776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:13.660416 26776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:13.660470 26776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:13.665887 26776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:13.666052 26776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:13.666076 26776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:13.666098 26776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:13.666115 26776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:13.674541 26776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:13.676242 26776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:13.677672 26776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:13.690667 26784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:13.690739 26781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:13.690862 26782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:13.690991 26776 server_base.cc:1034] running on GCE node
I20260923 18:06:13.691908 26776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:13.694324 26776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:13.695628 26776 hybrid_clock.cc:648] HybridClock initialized: now 1790186773695589 us; error 59 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T18:06:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1440 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:13 dist-test-slave-rvjj krb5kdc[26789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:13 dist-test-slave-rvjj krb5kdc[26789](info): set up 2 sockets
Sep 23 18:06:13 dist-test-slave-rvjj krb5kdc[26789](info): commencing operation
krb5kdc: starting...
W20260923 18:06:14.702306 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:14 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:06:14.850993 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36411
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:15.003997 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:15.004462 26802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:15.004530 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:15.009374 26802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:15.009490 26802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:15.009519 26802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:15.009542 26802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:15.009562 26802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:15.016423 26802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:15.018028 26802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:15.019483 26802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:15.031719 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.031770 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:15.031809 26802 server_base.cc:1034] running on GCE node
W20260923 18:06:15.031714 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:15.033015 26802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:15.034437 26802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:15.035784 26802 hybrid_clock.cc:648] HybridClock initialized: now 1790186775035678 us; error 78 us; skew 500 ppm
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:15.047395 26802 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:06:15.048759 26802 webserver.cc:460] Webserver started at http://127.18.44.254:42651/ using document root <none> and password file <none>
I20260923 18:06:15.049450 26802 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:15.049533 26802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:15.049775 26802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:15.052273 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "310d7dad3bf54d599b99736511ba37b5"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "0d33812fd8b46aa8940f0a60b2c9a436"
server_key_iv: "d73bda0355ffc54181f83efbb57a4786"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.052906 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "310d7dad3bf54d599b99736511ba37b5"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "0d33812fd8b46aa8940f0a60b2c9a436"
server_key_iv: "d73bda0355ffc54181f83efbb57a4786"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.057502 26802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:06:15.060755 26817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.062548 26802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:15.062724 26802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "310d7dad3bf54d599b99736511ba37b5"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "0d33812fd8b46aa8940f0a60b2c9a436"
server_key_iv: "d73bda0355ffc54181f83efbb57a4786"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.062846 26802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:15.082907 26802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:15.086891 26802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:15.087101 26802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:15.095134 26802 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:36411
I20260923 18:06:15.095201 26869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:36411 every 8 connection(s)
I20260923 18:06:15.096292 26802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:15.100183 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:15.100976 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26802
I20260923 18:06:15.101117 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:15.101401 18611 external_mini_cluster.cc:1428] Setting key 2719ab05f29e4082be25204a98e38e1c
I20260923 18:06:15.110893 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Bootstrap starting.
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:15.114665 26870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:15.115919 26870 log.cc:824] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:15.119336 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: No bootstrap required, opened a new log
I20260923 18:06:15.120986 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:15.103248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.18.44.254:36411)
0923 18:06:15.103894 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:15.103920 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:15.103984 (+ 64us) server_negotiation.cc:407] Connection header received
0923 18:06:15.104893 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:15.104927 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:15.105558 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:15.106220 (+ 662us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:15.107017 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.108739 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.110070 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.110993 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.111235 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:15.114389 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:15.114399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:15.114426 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:15.114479 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:15.116840 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.117514 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.117521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.117530 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.117701 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.118098 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.118101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.118103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.118727 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:15.118948 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:15.119338 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 18:06:15.119751 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":138,"threads_started":1}
I20260923 18:06:15.124907 26870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:15.125319 26870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:15.125375 26870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d7dad3bf54d599b99736511ba37b5, State: Initialized, Role: FOLLOWER
I20260923 18:06:15.125861 26870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:15.126024 26870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:15.126071 26870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:15.126157 26870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:15.127575 26870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:15.128028 26870 leader_election.cc:302] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310d7dad3bf54d599b99736511ba37b5; no voters:
I20260923 18:06:15.128412 26870 leader_election.cc:288] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:15.128760 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:15.129249 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 LEADER]: Becoming Leader. State: Replica: 310d7dad3bf54d599b99736511ba37b5, State: Running, Role: LEADER
I20260923 18:06:15.129948 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:15.130364 26870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:15.132197 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310d7dad3bf54d599b99736511ba37b5. Latest consensus state: current_term: 1 leader_uuid: "310d7dad3bf54d599b99736511ba37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } } }
I20260923 18:06:15.132411 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:15.132926 26883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:15.133226 26876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310d7dad3bf54d599b99736511ba37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } } }
I20260923 18:06:15.133368 26876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:15.136639 26883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:15.145033 26883 catalog_manager.cc:1347] Generated new cluster ID: 232207e261e547df8b15cbc182d9c63f
I20260923 18:06:15.145157 26883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:15.157974 26883 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:15.159466 26883 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:15.176222 26883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Generated new TSK 0
I20260923 18:06:15.177366 26883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:06:15.216171 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:15.355098 26896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:15.355470 26896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:15.355522 26896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:15.359876 26896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:15.359973 26896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:15.360036 26896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:15.366087 26896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:15.367394 26896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:15.368705 26896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:06:15.380991 26896 server_base.cc:1034] running on GCE node
W20260923 18:06:15.380975 26902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.380965 26904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.381024 26901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:15.382045 26896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:15.383049 26896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:15.384287 26896 hybrid_clock.cc:648] HybridClock initialized: now 1790186775384247 us; error 59 us; skew 500 ppm
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:15.388198 26896 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:06:15.389632 26896 webserver.cc:460] Webserver started at http://127.18.44.193:42759/ using document root <none> and password file <none>
I20260923 18:06:15.390458 26896 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:15.390565 26896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:15.390836 26896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:15.393616 26896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "58252f5f45fb445db09ea8c03af881a6"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "a597995a5eb934efab2365f728acf7b7"
server_key_iv: "fd1e162b62f0dfd5dbb4196b4a79a4b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.394424 26896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "58252f5f45fb445db09ea8c03af881a6"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "a597995a5eb934efab2365f728acf7b7"
server_key_iv: "fd1e162b62f0dfd5dbb4196b4a79a4b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.400683 26896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:06:15.405495 26911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.407778 26896 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:06:15.408022 26896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "58252f5f45fb445db09ea8c03af881a6"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "a597995a5eb934efab2365f728acf7b7"
server_key_iv: "fd1e162b62f0dfd5dbb4196b4a79a4b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.408193 26896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:15.438158 26896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:15.442502 26896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:15.442759 26896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:15.443645 26896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:15.445160 26896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:15.445236 26896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.445310 26896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:15.445339 26896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.457736 26896 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:40391
I20260923 18:06:15.457804 27024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:40391 every 8 connection(s)
I20260923 18:06:15.459005 26896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:15.466393 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 26896
I20260923 18:06:15.466516 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:15.466776 18611 external_mini_cluster.cc:1428] Setting key 8fbdb37074931ec581094fdd0286dd9d
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:06:15.477773 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:15.461614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:51121 (local address 127.18.44.254:36411)
0923 18:06:15.461840 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:15.461854 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:15.463331 (+ 1477us) server_negotiation.cc:407] Connection header received
0923 18:06:15.464517 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:15.464521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:15.464584 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:15.464666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:15.466203 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.466942 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.468362 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.468672 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.468796 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:15.472367 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:15.472371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:15.472374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:15.472413 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:15.474359 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.475230 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.475235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.475236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.475384 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.475992 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.475996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.475998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.476296 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:15.476417 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:15.477472 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0923 18:06:15.477621 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 18:06:15.479296 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:15.462127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:51121)
0923 18:06:15.463098 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:15.463159 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:15.464263 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:15.464890 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:15.464899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:15.465321 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:15.465927 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:15.465942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.467169 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.467173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:15.468203 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:15.468214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.469007 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.469013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:15.469255 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:15.470048 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:15.470068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:15.472099 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:15.474572 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:15.474581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:15.474619 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:15.475047 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:15.475576 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:15.475579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:15.475581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:15.475837 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:15.476566 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:15.476575 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:15.477070 (+ 495us) client_negotiation.cc:769] Sending connection context
0923 18:06:15.477345 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 18:06:15.477787 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":183,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:06:15.481371 27025 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36411
I20260923 18:06:15.481690 27025 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:15.482313 27025 heartbeater.cc:511] Master 127.18.44.254:36411 requested a full tablet report, sending...
I20260923 18:06:15.484978 26834 ts_manager.cc:194] Registered new tserver with Master: 58252f5f45fb445db09ea8c03af881a6 (127.18.44.193:40391)
I20260923 18:06:15.486970 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:51121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:06:15.503989 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:15.659343 27030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:15.659830 27030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:15.659888 27030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:15.664884 27030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:15.665011 27030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:15.665087 27030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:15.672132 27030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:15.673769 27030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:15.675283 27030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:15.689853 27036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.689885 27038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.689860 27035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:15.690546 27030 server_base.cc:1034] running on GCE node
I20260923 18:06:15.691198 27030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:15.692034 27030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:15.693382 27030 hybrid_clock.cc:648] HybridClock initialized: now 1790186775693343 us; error 71 us; skew 500 ppm
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:15.697106 27030 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:06:15.698220 27030 webserver.cc:460] Webserver started at http://127.18.44.194:44523/ using document root <none> and password file <none>
I20260923 18:06:15.698907 27030 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:15.698988 27030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:15.699231 27030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:15.701645 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "adf977b598f14ece84f1eea286b1949b"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "3110866fe76110f50db00037751f7111"
server_key_iv: "7ecf3a9781f793c2a84d9a37d29554d2"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.702306 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "adf977b598f14ece84f1eea286b1949b"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "3110866fe76110f50db00037751f7111"
server_key_iv: "7ecf3a9781f793c2a84d9a37d29554d2"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.707365 27030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 18:06:15.710876 27045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.712988 27030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 18:06:15.713178 27030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "adf977b598f14ece84f1eea286b1949b"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "3110866fe76110f50db00037751f7111"
server_key_iv: "7ecf3a9781f793c2a84d9a37d29554d2"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.713336 27030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:15.730592 27030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:15.734941 27030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:15.735150 27030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:15.735865 27030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:15.736824 27030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:15.736876 27030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.736933 27030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:15.736963 27030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:15.748346 27030 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39437
I20260923 18:06:15.748379 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39437 every 8 connection(s)
I20260923 18:06:15.749379 27030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:15.754746 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27030
I20260923 18:06:15.754894 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:15.755306 18611 external_mini_cluster.cc:1428] Setting key 1b3aac45cd4b3adf279a2a1d5f355b3b
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:06:15.768133 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:15.752086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:42461 (local address 127.18.44.254:36411)
0923 18:06:15.752225 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:15.752229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:15.753830 (+ 1601us) server_negotiation.cc:407] Connection header received
0923 18:06:15.755487 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:15.755491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:15.755576 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:15.755656 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:15.757212 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.757742 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.759210 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.759673 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.759870 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:15.763632 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:15.763635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:15.763638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:15.763668 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:15.765567 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.766340 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.766343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.766345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.766443 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:15.766813 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:15.766816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:15.766818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:15.767020 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:15.767112 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:15.767808 (+ 696us) server_negotiation.cc:299] Negotiation successful
0923 18:06:15.767949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:06:15.769166 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:15.752594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:42461)
0923 18:06:15.753455 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:15.753533 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:15.755170 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:15.755904 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:15.755914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:15.756428 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:15.757057 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:15.757081 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.757880 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.757883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:15.758944 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:15.758956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:15.759912 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:15.759916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:15.760155 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:15.761140 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:15.761163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:15.763386 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:15.765762 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:15.765771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:15.765794 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:15.766229 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:15.766545 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:15.766548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:15.766550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:15.766712 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:15.767140 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:15.767149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:15.767504 (+ 355us) client_negotiation.cc:769] Sending connection context
0923 18:06:15.767733 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 18:06:15.768120 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":224,"threads_started":1}
I20260923 18:06:15.770823 27159 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:06:15.771373 27159 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:15.772339 27159 heartbeater.cc:511] Master 127.18.44.254:36411 requested a full tablet report, sending...
I20260923 18:06:15.774513 26834 ts_manager.cc:194] Registered new tserver with Master: adf977b598f14ece84f1eea286b1949b (127.18.44.194:39437)
I20260923 18:06:15.775429 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:42461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:06:15.800217 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:15.949604 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:15.950009 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:15.950060 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:15.954780 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:15.954906 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:15.954977 27164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:15.962366 27164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:15.964314 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:15.965780 27164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:15.981508 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.981465 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:15.981472 27172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:15.982023 27164 server_base.cc:1034] running on GCE node
I20260923 18:06:15.982620 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:15.983469 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:15.984702 27164 hybrid_clock.cc:648] HybridClock initialized: now 1790186775984677 us; error 49 us; skew 500 ppm
Sep 23 18:06:15 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:15.988466 27164 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:06:15.989593 27164 webserver.cc:460] Webserver started at http://127.18.44.195:42013/ using document root <none> and password file <none>
I20260923 18:06:15.990293 27164 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:15.990370 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:15.990605 27164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:15.992981 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "6763c3b83dc84a848939210cb596ae74"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "f66b9ee38a10e29cb4d2775e8242bdce"
server_key_iv: "356c153ab60e272945fdeafd4955de35"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.993604 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "6763c3b83dc84a848939210cb596ae74"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "f66b9ee38a10e29cb4d2775e8242bdce"
server_key_iv: "356c153ab60e272945fdeafd4955de35"
server_key_version: "encryptionkey@0"
I20260923 18:06:15.998315 27164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:06:16.002290 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.004314 27164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260923 18:06:16.004491 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "6763c3b83dc84a848939210cb596ae74"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "f66b9ee38a10e29cb4d2775e8242bdce"
server_key_iv: "356c153ab60e272945fdeafd4955de35"
server_key_version: "encryptionkey@0"
I20260923 18:06:16.004634 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:16.035717 27164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:16.039794 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:16.040031 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:16.040690 27164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:16.041656 27164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:16.041709 27164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.041776 27164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:16.041805 27164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.053246 27164 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:35659
I20260923 18:06:16.053318 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:35659 every 8 connection(s)
I20260923 18:06:16.054399 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:16.060604 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27164
I20260923 18:06:16.060784 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:16.061116 18611 external_mini_cluster.cc:1428] Setting key dc41b4c9a03ac8b69ef85d74a86897e4
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:16.072013 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.057321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:37649 (local address 127.18.44.254:36411)
0923 18:06:16.057584 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.057589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.058710 (+ 1121us) server_negotiation.cc:407] Connection header received
0923 18:06:16.060673 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.060676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.060752 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.060883 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.062524 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.063048 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.064359 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.064579 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.064670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.068111 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:16.068116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:16.068119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:16.068159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:16.069680 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:16.070286 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:16.070289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:16.070290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:16.070372 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:16.070730 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:16.070732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:16.070733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:16.070968 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:16.071054 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:16.071688 (+ 634us) server_negotiation.cc:299] Negotiation successful
0923 18:06:16.071850 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260923 18:06:16.072991 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.057728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:37649)
0923 18:06:16.058466 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.058538 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.060270 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.061185 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.061194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.061668 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.062373 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.062398 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.063185 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.063188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.064216 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.064236 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.064744 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.064748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.065009 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.065949 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.065967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.067856 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:16.069837 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:16.069843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:16.069858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:16.070178 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:16.070455 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:16.070457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:16.070459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:16.070623 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:16.071068 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:16.071074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:16.071407 (+ 333us) client_negotiation.cc:769] Sending connection context
0923 18:06:16.071581 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:06:16.071957 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":146,"threads_started":1}
I20260923 18:06:16.074900 27293 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36411
I20260923 18:06:16.075225 27293 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:16.075955 27293 heartbeater.cc:511] Master 127.18.44.254:36411 requested a full tablet report, sending...
I20260923 18:06:16.077672 26834 ts_manager.cc:194] Registered new tserver with Master: 6763c3b83dc84a848939210cb596ae74 (127.18.44.195:35659)
I20260923 18:06:16.078486 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:37649
I20260923 18:06:16.090000 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
Entry for principal oryx/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab.
WARNING: no policy specified for oryx/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
Entry for principal oryx/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab.
WARNING: no policy specified for oryx/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.44.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
Entry for principal oryx/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab.
I20260923 18:06:16.183617 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26896
I20260923 18:06:16.190442 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27030
I20260923 18:06:16.196493 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27164
I20260923 18:06:16.202275 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 26802
I20260923 18:06:16.209160 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36411
--webserver_interface=127.18.44.254
--webserver_port=42651
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:16.331499 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:16.331827 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:16.331883 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:16.336377 27302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:16.336457 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:16.336484 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:16.336504 27302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:16.336525 27302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:16.342355 27302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=42651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:16.343605 27302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:16.344691 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:16.355242 27308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:16.355446 27302 server_base.cc:1034] running on GCE node
W20260923 18:06:16.355192 27307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:16.355202 27310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:16.356341 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:16.357699 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:16.358942 27302 hybrid_clock.cc:648] HybridClock initialized: now 1790186776358922 us; error 53 us; skew 500 ppm
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186776, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:16.362555 27302 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:06:16.363801 27302 webserver.cc:460] Webserver started at http://127.18.44.254:42651/ using document root <none> and password file <none>
I20260923 18:06:16.364415 27302 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:16.364501 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:16.369011 27302 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:06:16.371846 27317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.373544 27302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:06:16.373759 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "310d7dad3bf54d599b99736511ba37b5"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "0d33812fd8b46aa8940f0a60b2c9a436"
server_key_iv: "d73bda0355ffc54181f83efbb57a4786"
server_key_version: "encryptionkey@0"
I20260923 18:06:16.374341 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:16.392125 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:16.395810 27302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:16.396020 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:16.404114 27302 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:36411
I20260923 18:06:16.404158 27369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:36411 every 8 connection(s)
I20260923 18:06:16.405330 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:16.407578 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27302
I20260923 18:06:16.408118 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:40391
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:36411
--webserver_port=42759
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:06:16.412454 27370 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:06:16.415027 27370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Bootstrap starting.
I20260923 18:06:16.437358 27370 log.cc:824] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:16.446480 27370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:06:16.446976 27370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Bootstrap complete.
I20260923 18:06:16.453234 27370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:16.454036 27370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d7dad3bf54d599b99736511ba37b5, State: Initialized, Role: FOLLOWER
I20260923 18:06:16.454730 27370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:16.454893 27370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:16.454977 27370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:16.455084 27370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:06:16.458160 27370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:16.458679 27370 leader_election.cc:302] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310d7dad3bf54d599b99736511ba37b5; no voters:
I20260923 18:06:16.459116 27370 leader_election.cc:288] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:06:16.459295 27374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:06:16.459537 27374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [term 2 LEADER]: Becoming Leader. State: Replica: 310d7dad3bf54d599b99736511ba37b5, State: Running, Role: LEADER
I20260923 18:06:16.459959 27374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } }
I20260923 18:06:16.461021 27370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:16.461992 27375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "310d7dad3bf54d599b99736511ba37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } } }
I20260923 18:06:16.462183 27375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:16.462054 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310d7dad3bf54d599b99736511ba37b5. Latest consensus state: current_term: 2 leader_uuid: "310d7dad3bf54d599b99736511ba37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d7dad3bf54d599b99736511ba37b5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36411 } } }
I20260923 18:06:16.462283 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:16.462692 27380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:16.466958 27380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:16.467985 27380 catalog_manager.cc:1259] Loaded cluster ID: 232207e261e547df8b15cbc182d9c63f
I20260923 18:06:16.468068 27380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:16.470852 27380 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:16.471927 27380 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 310d7dad3bf54d599b99736511ba37b5: Loaded TSK: 0
I20260923 18:06:16.474081 27380 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:06:16.534319 27372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:16.534708 27372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:16.534771 27372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:16.539328 27372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:16.539426 27372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:16.539494 27372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:16.545403 27372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.193.keytab
--principal=oryx/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:40391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=42759
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:16.546617 27372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:16.547838 27372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:16.562140 27399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:16.562199 27397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:16.562132 27396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:16.563125 27372 server_base.cc:1034] running on GCE node
I20260923 18:06:16.563737 27372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:16.564507 27372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:16.565723 27372 hybrid_clock.cc:648] HybridClock initialized: now 1790186776565687 us; error 52 us; skew 500 ppm
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186776, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:16.569366 27372 init.cc:377] Logged in from keytab as oryx/127.18.44.193@KRBTEST.COM (short username oryx)
I20260923 18:06:16.570520 27372 webserver.cc:460] Webserver started at http://127.18.44.193:42759/ using document root <none> and password file <none>
I20260923 18:06:16.571171 27372 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:16.571300 27372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:16.576270 27372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260923 18:06:16.579671 27406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.581602 27372 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:06:16.581804 27372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "58252f5f45fb445db09ea8c03af881a6"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "a597995a5eb934efab2365f728acf7b7"
server_key_iv: "fd1e162b62f0dfd5dbb4196b4a79a4b7"
server_key_version: "encryptionkey@0"
I20260923 18:06:16.582456 27372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:16.601990 27372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:16.606045 27372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:16.606289 27372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:16.606952 27372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:16.608129 27372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:16.608191 27372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.608258 27372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:16.608297 27372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.619707 27372 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:40391
I20260923 18:06:16.619755 27519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:40391 every 8 connection(s)
I20260923 18:06:16.620857 27372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:16.626372 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27372
I20260923 18:06:16.627112 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:39437
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:36411
--webserver_port=44523
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:16.638554 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.624696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:59451)
0923 18:06:16.625762 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.625822 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.626981 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.628690 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.628700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.629381 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.629939 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.629957 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.631423 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.631426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.632372 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.632384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.633605 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.633609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.633776 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.634410 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.634428 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.637352 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":223,"threads_started":1}
W20260923 18:06:16.638537 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.624214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:59451 (local address 127.18.44.254:36411)
0923 18:06:16.624733 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.624751 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.626001 (+ 1250us) server_negotiation.cc:407] Connection header received
0923 18:06:16.627521 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.627532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.628198 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.628548 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.630159 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.631220 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.632525 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.633166 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.633344 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.637648 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:59451: BlockingRecv error: recv got EOF from 127.18.44.193:59451 (error 108)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":136,"threads_started":1}
W20260923 18:06:16.639482 27520 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.44.254:36411: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:16.645499 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.639995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:52753)
0923 18:06:16.640120 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.640135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.640370 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.640646 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.640649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.640837 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.640952 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.640958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.641752 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.641755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.642312 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.642320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.642786 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.642789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.642883 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.643437 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.643452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.645353 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 18:06:16.645655 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.640256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:52753 (local address 127.18.44.254:36411)
0923 18:06:16.640369 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.640379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.640397 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:06:16.640479 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.640482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.640542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.640607 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.641083 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.641630 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.642427 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.642662 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.642751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.645545 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:52753: BlockingRecv error: recv got EOF from 127.18.44.193:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:16.652269 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.646422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:54247)
0923 18:06:16.646750 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.646767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.646858 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.647239 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.647242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.647436 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.647523 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.647529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.648275 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.648278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.648823 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.648837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.649295 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.649298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.649380 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.649901 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.649924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.652117 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 18:06:16.652408 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.646570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:54247 (local address 127.18.44.254:36411)
0923 18:06:16.646991 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.646995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.647005 (+ 10us) server_negotiation.cc:407] Connection header received
0923 18:06:16.647041 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.647043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.647082 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.647158 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.647652 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.648162 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.648961 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.649177 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.649269 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.652298 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:54247: BlockingRecv error: recv got EOF from 127.18.44.193:54247 (error 108)
Metrics: {"server-negotiator.queue_time_us":344}
W20260923 18:06:16.652837 27520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:06:16.792074 27524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:16.792508 27524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:16.792578 27524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:16.797962 27524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:16.798089 27524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:16.798188 27524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:16.805531 27524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.194.keytab
--principal=oryx/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:39437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=44523
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:16.807191 27524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:16.808704 27524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:16.823560 27530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:16.823796 27529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:16.824026 27524 server_base.cc:1034] running on GCE node
W20260923 18:06:16.823560 27532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:16.824612 27524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:16.825843 27524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:16.827142 27524 hybrid_clock.cc:648] HybridClock initialized: now 1790186776827135 us; error 86 us; skew 500 ppm
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186776, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:16.831401 27524 init.cc:377] Logged in from keytab as oryx/127.18.44.194@KRBTEST.COM (short username oryx)
I20260923 18:06:16.832572 27524 webserver.cc:460] Webserver started at http://127.18.44.194:44523/ using document root <none> and password file <none>
I20260923 18:06:16.833253 27524 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:16.833351 27524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:16.838599 27524 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:16.841753 27539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.843958 27524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:06:16.844135 27524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "adf977b598f14ece84f1eea286b1949b"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "3110866fe76110f50db00037751f7111"
server_key_iv: "7ecf3a9781f793c2a84d9a37d29554d2"
server_key_version: "encryptionkey@0"
I20260923 18:06:16.844857 27524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:16.858002 27524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:16.861529 27524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:16.861717 27524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:16.862416 27524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:16.863474 27524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:16.863526 27524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.863583 27524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:16.863612 27524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:16.873992 27524 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39437
I20260923 18:06:16.874234 27652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39437 every 8 connection(s)
I20260923 18:06:16.875365 27524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:16.879110 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27524
I20260923 18:06:16.879644 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:35659
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:36411
--webserver_port=42013
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:16.892488 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.878198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:37091 (local address 127.18.44.254:36411)
0923 18:06:16.878345 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.878348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.879716 (+ 1368us) server_negotiation.cc:407] Connection header received
0923 18:06:16.881336 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.881341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.881427 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.881568 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.883795 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.884575 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.886540 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.886975 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.887153 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.892334 (+ 5181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:37091: BlockingRecv error: recv got EOF from 127.18.44.194:37091 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:06:16.893332 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.878706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:37091)
0923 18:06:16.879462 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.879542 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.881020 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.881865 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.881874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.882412 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.883207 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.883228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.884861 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.884866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.886262 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.886284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.887300 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.887306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.887629 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.888537 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.888568 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.892100 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":208,"threads_started":1}
W20260923 18:06:16.894176 27653 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.44.254:36411: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:16.902225 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.894822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:58655)
0923 18:06:16.895071 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.895090 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.895372 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.895778 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.895781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.895993 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.896146 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.896154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.897080 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.897083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.898061 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.898078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.898734 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.898738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.898849 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.899535 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.899550 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.902066 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260923 18:06:16.902468 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.895003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:58655 (local address 127.18.44.254:36411)
0923 18:06:16.895174 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.895180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.895297 (+ 117us) server_negotiation.cc:407] Connection header received
0923 18:06:16.895510 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.895583 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.895646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.895725 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.896327 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.896954 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.898251 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.898582 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.898697 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.902301 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:58655: BlockingRecv error: recv got EOF from 127.18.44.194:58655 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:16 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:16.909633 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:16.903110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:42407)
0923 18:06:16.903314 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:16.903342 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:16.903444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:16.903812 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:16.903815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:16.904017 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:16.904136 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.904143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.905041 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.905045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.905910 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:16.905923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.906503 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.906506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:16.906600 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.907207 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:16.907223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:16.909468 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 18:06:16.910133 27653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:06:16.910288 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:16.903211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:42407 (local address 127.18.44.254:36411)
0923 18:06:16.903407 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:16.903411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:16.903429 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:06:16.903616 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:16.903619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:16.903673 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:16.903760 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:16.904294 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.904912 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.906076 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:16.906355 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:16.906470 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:16.910127 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:42407: BlockingRecv error: recv got EOF from 127.18.44.194:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260923 18:06:17.013211 27656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:17.013541 27656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:17.013697 27656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:17.018537 27656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:17.018646 27656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:17.018733 27656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:17.024575 27656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/krb5kdc/oryx_127.18.44.195.keytab
--principal=oryx/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:35659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=42013
--tserver_master_addrs=127.18.44.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:17.025915 27656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:17.027240 27656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:17.039820 27662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:17.039853 27661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:17.039959 27656 server_base.cc:1034] running on GCE node
W20260923 18:06:17.039853 27664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:17.040779 27656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:17.041783 27656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:17.043018 27656 hybrid_clock.cc:648] HybridClock initialized: now 1790186777042981 us; error 54 us; skew 500 ppm
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186777, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:17.046941 27656 init.cc:377] Logged in from keytab as oryx/127.18.44.195@KRBTEST.COM (short username oryx)
I20260923 18:06:17.048323 27656 webserver.cc:460] Webserver started at http://127.18.44.195:42013/ using document root <none> and password file <none>
I20260923 18:06:17.049232 27656 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:17.049347 27656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:17.054842 27656 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 18:06:17.058379 27671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:17.060417 27656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260923 18:06:17.060729 27656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "6763c3b83dc84a848939210cb596ae74"
format_stamp: "Formatted at 2026-09-23 18:06:15 on dist-test-slave-rvjj"
server_key: "f66b9ee38a10e29cb4d2775e8242bdce"
server_key_iv: "356c153ab60e272945fdeafd4955de35"
server_key_version: "encryptionkey@0"
I20260923 18:06:17.061677 27656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:17.097025 27656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:17.101302 27656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:17.101534 27656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:17.102284 27656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:17.103492 27656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:17.103552 27656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:17.103605 27656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:17.103628 27656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:17.116386 27656 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:35659
I20260923 18:06:17.116478 27784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:35659 every 8 connection(s)
I20260923 18:06:17.117821 27656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:17.120711 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27656
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:17.132014 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.122004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.18.44.254:36411)
0923 18:06:17.122434 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.122438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.122454 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:17.122509 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.122512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.122563 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.122680 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.123629 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.124215 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.125162 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.125400 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.125495 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.127349 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:17.127356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:17.127383 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:17.127422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:17.129612 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:17.130066 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:17.130072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:17.130079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:17.130220 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:17.130490 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:17.130493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:17.130494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:17.130885 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:17.131096 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:17.131539 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 18:06:17.131825 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":97,"threads_started":1}
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:17.133622 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:17.120673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:46189 (local address 127.18.44.254:36411)
0923 18:06:17.120905 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.120911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.122583 (+ 1672us) server_negotiation.cc:407] Connection header received
0923 18:06:17.124105 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.124109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.124187 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.124273 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.125998 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.126754 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.128515 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.128842 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.128968 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.133510 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:46189: BlockingRecv error: recv got EOF from 127.18.44.195:46189 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 18:06:17.134402 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.121140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:46189)
0923 18:06:17.122256 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:17.122345 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:17.123784 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:17.124557 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:17.124567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:17.125099 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:17.125797 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.125817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.127042 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.127047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.128374 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.128393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.129153 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.129158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.129405 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.130221 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:17.130241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:17.133400 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":177,"threads_started":1}
W20260923 18:06:17.134908 27785 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.44.254:36411: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:17.141341 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.135499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:56827)
0923 18:06:17.135650 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:17.135669 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:17.135772 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:17.136050 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:17.136053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:17.136230 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:17.136332 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.136338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.137097 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.137100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.137656 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.137664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.138086 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.138090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.138178 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.138752 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:17.138764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:17.141179 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 18:06:17.141546 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:17.135591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:56827 (local address 127.18.44.254:36411)
0923 18:06:17.135712 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.135717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.135730 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:06:17.135896 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.135900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.135946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.136010 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.136446 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.136988 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.137764 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.137983 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.138066 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.141385 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:56827: BlockingRecv error: recv got EOF from 127.18.44.195:56827 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:17.148746 27522 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:17.142382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:53085 (local address 127.18.44.254:36411)
0923 18:06:17.142586 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.142590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.142601 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:06:17.142636 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.142638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.142673 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.142812 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.143231 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.143800 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.144553 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.144781 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.144869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.148580 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:53085: BlockingRecv error: recv got EOF from 127.18.44.195:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260923 18:06:17.149242 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.142230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:53085)
0923 18:06:17.142378 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:17.142393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:17.142483 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:17.142853 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:17.142856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:17.143033 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:17.143116 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.143121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.143904 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.143907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.144444 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.144452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.144889 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.144892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.144968 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.145607 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:17.145619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:17.149094 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 18:06:17.149717 27785 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:17.660481 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.653743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:53555)
0923 18:06:17.654101 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:17.654116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:17.654244 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:17.654669 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:17.654672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:17.654897 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:17.655066 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.655072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.656109 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.656113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.656819 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.656833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.657428 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.657432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.657586 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.658316 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:17.658333 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:17.660345 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":131,"threads_started":1}
W20260923 18:06:17.660877 27791 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:17.653926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:53555 (local address 127.18.44.254:36411)
0923 18:06:17.654270 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.654273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.654292 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:06:17.654344 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.654346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.654391 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.654523 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.655236 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.655904 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.656985 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.657274 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.657381 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.660700 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:53555: BlockingRecv error: recv got EOF from 127.18.44.193:53555 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:17 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:17.918798 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:17.911134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:49753)
0923 18:06:17.911618 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:17.911640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:17.911823 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:17.912153 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:17.912157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:17.912440 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:17.912585 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.912594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.913725 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.913730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.914643 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:17.914658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.915353 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.915357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:17.915442 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.916121 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:17.916136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:17.918599 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":140,"threads_started":1}
W20260923 18:06:17.919054 27791 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:17.911365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:49753 (local address 127.18.44.254:36411)
0923 18:06:17.911612 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:17.911619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:17.911754 (+ 135us) server_negotiation.cc:407] Connection header received
0923 18:06:17.911935 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:17.911938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:17.912018 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:17.912112 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:17.912868 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.913547 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.914891 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:17.915138 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:17.915241 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:17.918891 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:49753: BlockingRecv error: recv got EOF from 127.18.44.194:49753 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:18.157027 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:18.150508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:50901)
0923 18:06:18.150784 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:18.150797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:18.150890 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:18.151275 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:18.151278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:18.151464 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:18.151571 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.151577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.152571 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.152573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.153229 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.153238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.153969 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.153973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.154065 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.154668 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:18.154689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:18.156888 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
W20260923 18:06:18.157346 27791 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:18.150683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50901 (local address 127.18.44.254:36411)
0923 18:06:18.150845 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:18.150850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:18.150871 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:06:18.151014 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:18.151019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:18.151104 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:18.151215 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:18.151725 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.152349 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.153405 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.153808 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.153936 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.157178 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:50901: BlockingRecv error: recv got EOF from 127.18.44.195:50901 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:18.668701 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:18.661464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:60055)
0923 18:06:18.661905 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:18.661925 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:18.662091 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:18.662478 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:18.662482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:18.662690 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:18.662809 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.662816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.663721 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.663725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.664356 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.664367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.664817 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.664820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.664903 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.665611 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:18.665627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:18.668507 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":85,"threads_started":1}
W20260923 18:06:18.669021 27795 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:18.661570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:60055 (local address 127.18.44.254:36411)
0923 18:06:18.661951 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:18.661956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:18.662011 (+ 55us) server_negotiation.cc:407] Connection header received
0923 18:06:18.662241 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:18.662244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:18.662323 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:18.662425 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:18.662972 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.663604 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.664467 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.664707 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.664798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.668834 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:60055: BlockingRecv error: recv got EOF from 127.18.44.193:60055 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:18 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:18.933372 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:18.920064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:43965)
0923 18:06:18.920538 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:18.920555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:18.920692 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:18.921107 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:18.921110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:18.921339 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:18.921514 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.921521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.922699 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.922705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.923521 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:18.923538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.924192 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.924196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:18.924286 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.930634 (+ 6348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:18.930655 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:18.933193 (+ 2538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
W20260923 18:06:18.933686 27795 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:18.920242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:43965 (local address 127.18.44.254:36411)
0923 18:06:18.920395 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:18.920399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:18.920664 (+ 265us) server_negotiation.cc:407] Connection header received
0923 18:06:18.920866 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:18.920869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:18.920931 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:18.921049 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:18.921693 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.922420 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.923726 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:18.924014 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:18.924124 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:18.933514 (+ 9390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:43965: BlockingRecv error: recv got EOF from 127.18.44.194:43965 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:19.165380 27795 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:19.158545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:33095 (local address 127.18.44.254:36411)
0923 18:06:19.158762 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:19.158767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:19.158786 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:06:19.158874 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:19.158876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:19.158936 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:19.159024 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:19.159762 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.160326 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.161345 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.161586 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.161697 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.165216 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:33095: BlockingRecv error: recv got EOF from 127.18.44.195:33095 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 18:06:19.165867 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:19.158158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:33095)
0923 18:06:19.158587 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:19.158610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:19.158780 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:19.159073 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:19.159077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:19.159385 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:19.159536 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.159543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.160483 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.160487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.161209 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.161222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.161761 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.161766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.161873 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.162488 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:19.162500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:19.165689 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":104,"threads_started":1}
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:19.676623 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:19.669860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:41049)
0923 18:06:19.670294 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:19.670312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:19.670466 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:19.670837 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:19.670840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:19.671033 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:19.671176 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.671182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.672157 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.672161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.672885 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.672898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.673456 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.673459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.673550 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.674187 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:19.674203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:19.676468 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":119,"threads_started":1}
W20260923 18:06:19.676956 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:19.670039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:41049 (local address 127.18.44.254:36411)
0923 18:06:19.670313 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:19.670318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:19.670441 (+ 123us) server_negotiation.cc:407] Connection header received
0923 18:06:19.670570 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:19.670573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:19.670631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:19.670734 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:19.671365 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.672012 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.673012 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.673303 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.673463 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.676777 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:41049: BlockingRecv error: recv got EOF from 127.18.44.193:41049 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:19 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:19.941148 27800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:19.934495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:47269)
0923 18:06:19.934863 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:19.934884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:19.935048 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:19.935374 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:19.935377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:19.935595 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:19.935709 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.935716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.936473 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.936476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.937009 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:19.937017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.937473 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.937477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:19.937554 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.938198 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:19.938209 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:19.940940 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
W20260923 18:06:19.941256 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:19.934707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:47269 (local address 127.18.44.254:36411)
0923 18:06:19.934888 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:19.934894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:19.934988 (+ 94us) server_negotiation.cc:407] Connection header received
0923 18:06:19.935178 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:19.935181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:19.935233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:19.935325 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:19.935831 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.936366 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.937131 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:19.937358 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:19.937445 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:19.941118 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:47269: BlockingRecv error: recv got EOF from 127.18.44.194:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:20.186556 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:20.166921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:44045)
0923 18:06:20.167371 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:20.167400 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:20.167587 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:20.167973 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:20.167976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:20.168206 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:20.168358 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.168366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.169395 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.169399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.170434 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.170451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.171078 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.171082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.171170 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.183904 (+ 12734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:20.183925 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:20.186384 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":77,"threads_started":1}
W20260923 18:06:20.186717 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:20.167107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:44045 (local address 127.18.44.254:36411)
0923 18:06:20.167342 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:20.167346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:20.167522 (+ 176us) server_negotiation.cc:407] Connection header received
0923 18:06:20.167715 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:20.167718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:20.167778 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:20.167854 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:20.168501 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.169238 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.170610 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.170897 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.171006 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.186594 (+ 15588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:44045: BlockingRecv error: recv got EOF from 127.18.44.195:44045 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:20.685737 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:20.677888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:59605)
0923 18:06:20.678232 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:20.678249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:20.678392 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:20.678909 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:20.678912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:20.679291 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:20.679444 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.679453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.680503 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.680507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.681161 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.681173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.682091 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.682097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.682201 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.683110 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:20.683126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:20.685540 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
W20260923 18:06:20.686058 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:20.678029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:59605 (local address 127.18.44.254:36411)
0923 18:06:20.678212 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:20.678217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:20.678327 (+ 110us) server_negotiation.cc:407] Connection header received
0923 18:06:20.678601 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:20.678604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:20.678670 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:20.678824 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:20.679658 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.680234 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.681410 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.681804 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.682006 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.685900 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:59605: BlockingRecv error: recv got EOF from 127.18.44.193:59605 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:20 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:20.949072 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:20.942260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:51555)
0923 18:06:20.942568 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:20.942584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:20.942698 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:20.942991 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:20.942994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:20.943246 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:20.943426 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.943435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.944523 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.944527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.945299 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:20.945310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.945926 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.945929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:20.945996 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.946511 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:20.946523 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:20.948911 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
W20260923 18:06:20.949184 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:20.942557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:51555 (local address 127.18.44.254:36411)
0923 18:06:20.942770 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:20.942775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:20.942792 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:06:20.942836 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:20.942839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:20.942879 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:20.942949 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:20.943630 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.944338 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.945519 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:20.945785 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:20.945896 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:20.949048 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:51555: BlockingRecv error: recv got EOF from 127.18.44.194:51555 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:21.193912 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:21.187851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:39143)
0923 18:06:21.188217 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:21.188233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:21.188426 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:21.188689 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:21.188692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:21.188907 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:21.189008 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.189014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.189846 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.189849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.190428 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.190437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.190928 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.190932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.191014 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.191578 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:21.191591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:21.193708 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
W20260923 18:06:21.194219 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:21.187949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:39143 (local address 127.18.44.254:36411)
0923 18:06:21.188134 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:21.188140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:21.188362 (+ 222us) server_negotiation.cc:407] Connection header received
0923 18:06:21.188528 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:21.188531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:21.188582 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:21.188650 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:21.189132 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.189734 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.190551 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.190775 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.190879 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.194038 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:39143: BlockingRecv error: recv got EOF from 127.18.44.195:39143 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:21.694813 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:21.686946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:34913)
0923 18:06:21.687583 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:21.687605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:21.687815 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:21.688175 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:21.688178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:21.688491 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:21.688636 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.688644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.689552 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.689557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.690283 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.690298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.690942 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.690947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.691063 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.691796 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:21.691816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:21.694631 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":109,"threads_started":1}
W20260923 18:06:21.695079 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:21.687196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:34913 (local address 127.18.44.254:36411)
0923 18:06:21.687542 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:21.687547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:21.687790 (+ 243us) server_negotiation.cc:407] Connection header received
0923 18:06:21.687954 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:21.687957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:21.688014 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:21.688122 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:21.688780 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.689390 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.690441 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.690688 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.690809 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.694910 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:34913: BlockingRecv error: recv got EOF from 127.18.44.193:34913 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:21 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:21.956614 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:21.950082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:43367)
0923 18:06:21.950350 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:21.950367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:21.950485 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:21.950840 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:21.950844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:21.951050 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:21.951144 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.951150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.951980 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.951984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.952535 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:21.952543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.952985 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.952988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:21.953065 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.953827 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:21.953839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:21.956355 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
W20260923 18:06:21.956660 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:21.950276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:43367 (local address 127.18.44.254:36411)
0923 18:06:21.950418 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:21.950423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:21.950439 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:06:21.950588 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:21.950592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:21.950637 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:21.950809 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:21.951334 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.951867 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.952653 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:21.952875 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:21.952960 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:21.956544 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:43367: BlockingRecv error: recv got EOF from 127.18.44.194:43367 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:22.201581 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:22.195198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:37823)
0923 18:06:22.195646 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:22.195664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:22.195897 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:22.196156 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:22.196158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:22.196375 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:22.196472 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.196478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.197222 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.197226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.197793 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.197803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.198236 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.198240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.198316 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.198965 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:22.198978 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:22.201407 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":118,"threads_started":1}
W20260923 18:06:22.201732 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:22.195394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:37823 (local address 127.18.44.254:36411)
0923 18:06:22.195613 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:22.195618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:22.195847 (+ 229us) server_negotiation.cc:407] Connection header received
0923 18:06:22.195998 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:22.196001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:22.196047 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:22.196117 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:22.196598 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.197115 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.197912 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.198123 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.198210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.201609 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:37823: BlockingRecv error: recv got EOF from 127.18.44.195:37823 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:22.703485 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:22.696144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:56605)
0923 18:06:22.696447 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:22.696462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:22.696572 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:22.697036 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:22.697039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:22.697241 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:22.697366 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.697373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.698350 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.698353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.699013 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.699031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.699706 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.699709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.699791 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.700407 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:22.700419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:22.703294 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
W20260923 18:06:22.703743 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:22.696404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:56605 (local address 127.18.44.254:36411)
0923 18:06:22.696666 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:22.696671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:22.696695 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:06:22.696756 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:22.696760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:22.696815 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:22.696928 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:22.697554 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.698165 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.699235 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.699554 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.699665 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.703554 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:56605: BlockingRecv error: recv got EOF from 127.18.44.193:56605 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:22 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:22.964517 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:22.957893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:37811)
0923 18:06:22.958263 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:22.958286 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:22.958438 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:22.958715 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:22.958718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:22.958952 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:22.959072 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.959080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.960269 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.960273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.961090 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:22.961103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.961592 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.961596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:22.961693 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.962389 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:22.962402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:22.964378 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
W20260923 18:06:22.964851 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:22.958144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:37811 (local address 127.18.44.254:36411)
0923 18:06:22.958355 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:22.958360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:22.958378 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:06:22.958551 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:22.958554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:22.958602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:22.958672 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:22.959299 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.960123 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.961221 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:22.961473 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:22.961566 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:22.964674 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:37811: BlockingRecv error: recv got EOF from 127.18.44.194:37811 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:23.210031 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:23.202851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:47309)
0923 18:06:23.203329 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:23.203352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:23.203570 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:23.203967 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:23.203970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:23.204176 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:23.204289 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.204296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.205088 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.205092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.205716 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.205729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.206559 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.206562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.206672 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.207502 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:23.207516 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:23.209641 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":89,"threads_started":1}
W20260923 18:06:23.210206 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:23.203054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:47309 (local address 127.18.44.254:36411)
0923 18:06:23.203381 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:23.203387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:23.203484 (+ 97us) server_negotiation.cc:407] Connection header received
0923 18:06:23.203729 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:23.203733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:23.203798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:23.203905 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:23.204430 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.204976 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.206015 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.206362 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.206484 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.210074 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:47309: BlockingRecv error: recv got EOF from 127.18.44.195:47309 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:23.711678 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:23.704800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:35877)
0923 18:06:23.705427 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:23.705443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:23.705631 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:23.705943 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:23.705945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:23.706172 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:23.706297 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.706303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.707134 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.707138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.707878 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.707890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.708414 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.708417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.708504 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.709136 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:23.709148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:23.711537 (+ 2389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":163,"threads_started":1}
W20260923 18:06:23.712065 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:23.705056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35877 (local address 127.18.44.254:36411)
0923 18:06:23.705222 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:23.705227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:23.705527 (+ 300us) server_negotiation.cc:407] Connection header received
0923 18:06:23.705716 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:23.705718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:23.705829 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:23.705905 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:23.706396 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.706980 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.708033 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.708260 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.708356 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.711901 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:35877: BlockingRecv error: recv got EOF from 127.18.44.193:35877 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:23 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:23.973047 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:23.965838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:47481)
0923 18:06:23.966279 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:23.966298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:23.966549 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:23.966919 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:23.966922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:23.967160 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:23.967362 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.967372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.968344 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.968347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.969074 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:23.969086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.969625 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.969629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:23.969709 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.970268 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:23.970286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:23.972869 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":141,"threads_started":1}
W20260923 18:06:23.973198 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:23.965902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:47481 (local address 127.18.44.254:36411)
0923 18:06:23.966111 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:23.966116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:23.966412 (+ 296us) server_negotiation.cc:407] Connection header received
0923 18:06:23.966698 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:23.966701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:23.966758 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:23.966863 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:23.967539 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.968198 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.969223 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:23.969492 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:23.969604 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:23.973054 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:47481: BlockingRecv error: recv got EOF from 127.18.44.194:47481 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:24.218313 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:24.211327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:42879)
0923 18:06:24.212025 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:24.212045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:24.212213 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:24.212604 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:24.212608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:24.212868 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:24.212998 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.213006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.213820 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.213824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.214417 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.214427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.214905 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.214908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.215001 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.215684 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:24.215697 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:24.218147 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":229,"threads_started":1}
W20260923 18:06:24.218623 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:24.211403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:42879 (local address 127.18.44.254:36411)
0923 18:06:24.211556 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:24.211560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:24.212158 (+ 598us) server_negotiation.cc:407] Connection header received
0923 18:06:24.212319 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:24.212323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:24.212387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:24.212586 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:24.213129 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.213703 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.214552 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.214776 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.214878 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.218472 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:42879: BlockingRecv error: recv got EOF from 127.18.44.195:42879 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:24.719753 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:24.712954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:55489)
0923 18:06:24.713353 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:24.713375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:24.713540 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:24.713856 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:24.713859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:24.714126 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:24.714248 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.714254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.715068 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.715071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.715729 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.715742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.716226 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.716230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.716320 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.716922 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:24.716934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:24.719509 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
W20260923 18:06:24.719982 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:24.713290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:55489 (local address 127.18.44.254:36411)
0923 18:06:24.713452 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:24.713458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:24.713559 (+ 101us) server_negotiation.cc:407] Connection header received
0923 18:06:24.713665 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:24.713669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:24.713727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:24.713805 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:24.714376 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.714944 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.715855 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.716087 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.716184 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.719832 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:55489: BlockingRecv error: recv got EOF from 127.18.44.193:55489 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:24 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:24.981211 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:24.974127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:44995)
0923 18:06:24.974618 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:24.974639 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:24.974820 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:24.975123 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:24.975126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:24.975413 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:24.975544 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.975552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.976509 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.976512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.977101 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:24.977110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.977561 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.977564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:24.977645 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.978314 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:24.978330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:24.981062 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
W20260923 18:06:24.981624 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:24.974379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:44995 (local address 127.18.44.254:36411)
0923 18:06:24.974618 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:24.974624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:24.974745 (+ 121us) server_negotiation.cc:407] Connection header received
0923 18:06:24.974929 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:24.974932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:24.974983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:24.975070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:24.975680 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.976394 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.977214 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:24.977433 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:24.977527 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:24.981364 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:44995: BlockingRecv error: recv got EOF from 127.18.44.194:44995 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:25.231870 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:25.219692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:53545)
0923 18:06:25.220355 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:25.220375 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:25.220540 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:25.220852 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:25.220855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:25.221104 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:25.221238 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.221245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.222052 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.222055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.222656 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.222666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.223128 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.223132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.223212 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.229291 (+ 6079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:25.229312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:25.231693 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":110,"threads_started":1}
W20260923 18:06:25.232159 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:25.219781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:53545 (local address 127.18.44.254:36411)
0923 18:06:25.220013 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:25.220019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:25.220503 (+ 484us) server_negotiation.cc:407] Connection header received
0923 18:06:25.220670 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:25.220674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:25.220728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:25.220801 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:25.221362 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.221934 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.222781 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.223010 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.223104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.231991 (+ 8887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:53545: BlockingRecv error: recv got EOF from 127.18.44.195:53545 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:25.728291 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:25.720892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:38229)
0923 18:06:25.721250 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:25.721264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:25.721410 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:25.721756 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:25.721760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:25.721996 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:25.722129 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.722135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.723194 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.723200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.724036 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.724054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.724779 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.724783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.724869 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.725496 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:25.725509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:25.728109 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
W20260923 18:06:25.728418 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:25.721120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38229 (local address 127.18.44.254:36411)
0923 18:06:25.721272 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:25.721276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:25.721388 (+ 112us) server_negotiation.cc:407] Connection header received
0923 18:06:25.721501 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:25.721504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:25.721558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:25.721673 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:25.722281 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.722960 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.724244 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.724491 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.724666 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.728287 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:38229: BlockingRecv error: recv got EOF from 127.18.44.193:38229 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:25 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:25.989012 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:25.982317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:54749)
0923 18:06:25.982838 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:25.982853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:25.982994 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:25.983437 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:25.983440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:25.983676 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:25.983815 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.983821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.984671 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.984675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.985334 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:25.985347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.985868 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.985872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:25.985949 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.986590 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:25.986602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:25.988850 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":155,"threads_started":1}
W20260923 18:06:25.989230 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:25.982556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:54749 (local address 127.18.44.254:36411)
0923 18:06:25.982790 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:25.982795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:25.982958 (+ 163us) server_negotiation.cc:407] Connection header received
0923 18:06:25.983087 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:25.983090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:25.983144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:25.983316 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:25.983966 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.984502 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.985456 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:25.985676 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:25.985854 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:25.989075 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:54749: BlockingRecv error: recv got EOF from 127.18.44.194:54749 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:26.239404 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:26.233239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:50257)
0923 18:06:26.233546 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:26.233563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:26.233685 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:26.234024 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:26.234027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:26.234260 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:26.234361 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.234367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.235112 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.235116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.235714 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.235723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.236178 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.236181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.236259 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.236873 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:26.236885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:26.239168 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
W20260923 18:06:26.239629 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:26.233507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50257 (local address 127.18.44.254:36411)
0923 18:06:26.233801 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:26.233808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:26.233822 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:06:26.233863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:26.233866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:26.233906 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:26.233973 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:26.234487 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.235005 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.235834 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.236044 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.236152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.239503 (+ 3351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:50257: BlockingRecv error: recv got EOF from 127.18.44.195:50257 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:26.737425 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:26.729415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:40727)
0923 18:06:26.729873 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:26.729893 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:26.730082 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:26.730619 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:26.730622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:26.730878 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:26.731029 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.731038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.732072 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.732076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.733115 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.733140 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.733787 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.733790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.733889 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.734714 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:26.734731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:26.737175 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":109,"threads_started":1}
W20260923 18:06:26.737522 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:26.729685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:40727 (local address 127.18.44.254:36411)
0923 18:06:26.729833 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:26.729838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:26.730055 (+ 217us) server_negotiation.cc:407] Connection header received
0923 18:06:26.730255 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:26.730258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:26.730311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:26.730406 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:26.731215 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.731891 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.733349 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.733648 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.733758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.737405 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:40727: BlockingRecv error: recv got EOF from 127.18.44.193:40727 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:26 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:26.997716 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:26.990358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:47495)
0923 18:06:26.990783 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:26.990870 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:26.991156 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:26.991576 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:26.991579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:26.991840 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:26.991964 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.991970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.992823 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.992827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.993430 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:26.993442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.993950 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.993955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:26.994050 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.994789 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:26.994803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:26.997519 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
W20260923 18:06:26.997817 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:26.990653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:47495 (local address 127.18.44.254:36411)
0923 18:06:26.990835 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:26.990840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:26.990983 (+ 143us) server_negotiation.cc:407] Connection header received
0923 18:06:26.991356 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:26.991359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:26.991425 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:26.991506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:26.992113 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.992689 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.993583 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:26.993817 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:26.993919 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:26.997693 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:47495: BlockingRecv error: recv got EOF from 127.18.44.194:47495 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 18:06:27 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:27 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:27.247503 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:27.240557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:34683)
0923 18:06:27.240900 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:27.240916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:27.241044 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:27.241373 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:27.241375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:27.241590 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:27.241815 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:27.241822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.242766 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.242770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:27.243474 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:27.243491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.244059 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.244062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:27.244145 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:27.244840 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:27.244852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:27.247248 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
W20260923 18:06:27.247745 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:27.240708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:34683 (local address 127.18.44.254:36411)
0923 18:06:27.240871 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:27.240876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:27.241057 (+ 181us) server_negotiation.cc:407] Connection header received
0923 18:06:27.241166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:27.241170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:27.241232 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:27.241307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:27.241978 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.242625 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.243652 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.243917 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.244026 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:27.247508 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:34683: BlockingRecv error: recv got EOF from 127.18.44.195:34683 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 18:06:27 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:27 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:27.746013 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:27.738801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:45539)
0923 18:06:27.739241 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:27.739294 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:27.739430 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:27.739902 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:27.739906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:27.740118 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:27.740257 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:27.740263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.741310 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.741313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:27.741965 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:27.741976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.742563 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.742566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:27.742659 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:27.743320 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:27.743333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:27.745844 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":135,"threads_started":1}
W20260923 18:06:27.746387 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:27.739030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:45539 (local address 127.18.44.254:36411)
0923 18:06:27.739206 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:27.739212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:27.739378 (+ 166us) server_negotiation.cc:407] Connection header received
0923 18:06:27.739630 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:27.739634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:27.739718 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:27.739813 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:27.740503 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.741122 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.742127 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:27.742394 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:27.742578 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:27.746212 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:45539: BlockingRecv error: recv got EOF from 127.18.44.193:45539 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:28.006701 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:27.998967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:49575)
0923 18:06:27.999335 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:27.999350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:27.999465 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:28.000072 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:28.000075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:28.000288 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:28.000421 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.000428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.001555 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.001559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.002199 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.002212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.002945 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.002948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.003029 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.003766 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:28.003780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:28.006461 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":77,"threads_started":1}
W20260923 18:06:28.006939 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:27.999149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:49575 (local address 127.18.44.254:36411)
0923 18:06:27.999585 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:27.999589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:27.999613 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:06:27.999674 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:27.999677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:27.999845 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:27.999980 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:28.000611 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.001284 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.002490 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.002758 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.002887 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.006781 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:49575: BlockingRecv error: recv got EOF from 127.18.44.194:49575 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:28.256440 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:28.248857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:38853)
0923 18:06:28.249319 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:28.249341 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:28.249497 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:28.249929 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:28.249932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:28.250133 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:28.250270 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.250277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.251378 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.251382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.252077 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.252091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.252869 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.252873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.252951 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.253584 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:28.253597 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:28.256242 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":119,"threads_started":1}
W20260923 18:06:28.256518 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:28.248997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:38853 (local address 127.18.44.254:36411)
0923 18:06:28.249306 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:28.249312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:28.249554 (+ 242us) server_negotiation.cc:407] Connection header received
0923 18:06:28.249630 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:28.249633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:28.249698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:28.249814 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:28.250457 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.251128 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.252292 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.252632 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.252761 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.256395 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:38853: BlockingRecv error: recv got EOF from 127.18.44.195:38853 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:28 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:28.754292 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:28.746997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:57687)
0923 18:06:28.747435 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:28.747451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:28.747593 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:28.747975 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:28.747977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:28.748208 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:28.748349 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.748357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.749377 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.749381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.750108 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:28.750120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.750594 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.750597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:28.750672 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.751384 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:28.751402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:28.754111 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":119,"threads_started":1}
W20260923 18:06:28.754397 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:28.747120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:57687 (local address 127.18.44.254:36411)
0923 18:06:28.747365 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:28.747369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:28.747566 (+ 197us) server_negotiation.cc:407] Connection header received
0923 18:06:28.747691 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:28.747693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:28.747755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:28.747853 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:28.748537 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.749244 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.750232 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:28.750461 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:28.750551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:28.754284 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:57687: BlockingRecv error: recv got EOF from 127.18.44.193:57687 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:06:29.015019 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:29.008171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:49985 (local address 127.18.44.254:36411)
0923 18:06:29.008411 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:29.008417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:29.008573 (+ 156us) server_negotiation.cc:407] Connection header received
0923 18:06:29.008805 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:29.008809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:29.008854 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:29.008923 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:29.009385 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.009994 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.010810 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.011025 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.011117 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.014893 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:49985: BlockingRecv error: recv got EOF from 127.18.44.194:49985 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260923 18:06:29.015031 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:29.008005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:49985)
0923 18:06:29.008465 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:29.008482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:29.008699 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:29.008966 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:29.008969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:29.009165 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:29.009266 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.009272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.010145 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.010148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.010701 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.010709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.011141 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.011144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.011219 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.011870 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:29.011883 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:29.014827 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":150,"threads_started":1}
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:29.264791 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:29.257512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:52857)
0923 18:06:29.257915 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:29.257935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:29.258148 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:29.258528 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:29.258532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:29.258794 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:29.258918 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.258926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.259951 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.259954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.260581 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.260592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.261172 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.261176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.261266 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.261996 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:29.262014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:29.264576 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
W20260923 18:06:29.264909 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:29.257667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:52857 (local address 127.18.44.254:36411)
0923 18:06:29.257878 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:29.257884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:29.258044 (+ 160us) server_negotiation.cc:407] Connection header received
0923 18:06:29.258256 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:29.258260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:29.258312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:29.258479 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:29.259054 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.259755 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.260708 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.261043 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.261142 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.264780 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:52857: BlockingRecv error: recv got EOF from 127.18.44.195:52857 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:29 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:29.762081 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:29.755371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:41931)
0923 18:06:29.755744 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:29.755760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:29.755922 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:29.756245 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:29.756248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:29.756517 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:29.756638 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.756645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.757460 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.757463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.758114 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:29.758124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.758642 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.758646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:29.758738 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.759398 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:29.759411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:29.761939 (+ 2528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
W20260923 18:06:29.762414 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:29.755443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:41931 (local address 127.18.44.254:36411)
0923 18:06:29.755648 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:29.755653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:29.755859 (+ 206us) server_negotiation.cc:407] Connection header received
0923 18:06:29.756071 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:29.756074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:29.756123 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:29.756197 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:29.756765 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.757345 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.758240 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:29.758503 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:29.758603 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:29.762239 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:41931: BlockingRecv error: recv got EOF from 127.18.44.193:41931 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:30.024066 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:30.016112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:41609)
0923 18:06:30.016527 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:30.016546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:30.016714 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:30.017178 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:30.017181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:30.017456 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:30.017629 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.017636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.018486 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.018491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.019086 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.019097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.019639 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.019643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.019728 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.020354 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:30.020367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:30.023876 (+ 3509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":99,"threads_started":1}
W20260923 18:06:30.024248 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:30.016400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:41609 (local address 127.18.44.254:36411)
0923 18:06:30.016549 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:30.016555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:30.016634 (+ 79us) server_negotiation.cc:407] Connection header received
0923 18:06:30.016821 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:30.016825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:30.016888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:30.017159 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:30.017777 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.018353 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.019225 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.019505 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.019611 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.024120 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:41609: BlockingRecv error: recv got EOF from 127.18.44.194:41609 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:30.278405 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:30.265958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:49325)
0923 18:06:30.266515 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:30.266532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:30.266736 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:30.267041 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:30.267044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:30.267355 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:30.267487 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.267495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.268420 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.268424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.269271 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.269285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.270042 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.270046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.270135 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.275917 (+ 5782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:30.275936 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:30.278263 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":215,"threads_started":1}
W20260923 18:06:30.278829 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:30.266173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:49325 (local address 127.18.44.254:36411)
0923 18:06:30.266470 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:30.266476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:30.266638 (+ 162us) server_negotiation.cc:407] Connection header received
0923 18:06:30.266846 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:30.266850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:30.266914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:30.266994 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:30.267621 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.268276 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.269445 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.269867 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.270003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.278631 (+ 8628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:49325: BlockingRecv error: recv got EOF from 127.18.44.195:49325 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:30 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:30.770310 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:30.763584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:52397)
0923 18:06:30.763975 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:30.763995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:30.764172 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:30.764447 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:30.764450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:30.764753 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:30.764866 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.764874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.765718 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.765722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.766445 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:30.766457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.766954 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.766959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:30.767043 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.767644 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:30.767658 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:30.770139 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
W20260923 18:06:30.770506 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:30.763584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:52397 (local address 127.18.44.254:36411)
0923 18:06:30.763942 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:30.763948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:30.764198 (+ 250us) server_negotiation.cc:407] Connection header received
0923 18:06:30.764280 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:30.764284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:30.764332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:30.764410 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:30.765000 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.765602 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.766569 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:30.766791 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:30.766887 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:30.770339 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:52397: BlockingRecv error: recv got EOF from 127.18.44.193:52397 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:31.033893 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:31.025639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:40703)
0923 18:06:31.026109 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:31.026123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:31.026262 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:31.026747 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:31.026753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:31.027019 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:31.027210 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.027222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.028568 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.028573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.029362 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.029377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.030080 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.030083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.030306 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.030949 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:31.030965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:31.033645 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":110,"threads_started":1}
W20260923 18:06:31.034205 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:31.025707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:40703 (local address 127.18.44.254:36411)
0923 18:06:31.025961 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:31.025967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:31.026210 (+ 243us) server_negotiation.cc:407] Connection header received
0923 18:06:31.026362 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:31.026366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:31.026525 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:31.026626 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:31.027451 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.028386 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.029556 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.029825 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.030027 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.033940 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:40703: BlockingRecv error: recv got EOF from 127.18.44.194:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.195@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:31.286700 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:31.279575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.195:43429)
0923 18:06:31.279974 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:31.279996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:31.280199 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:31.280589 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:31.280593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:31.280921 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:31.281059 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.281067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.282202 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.282206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.282859 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.282869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.283524 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.283527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.283609 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.284325 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:31.284338 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:31.286523 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
W20260923 18:06:31.286990 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:31.279852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:43429 (local address 127.18.44.254:36411)
0923 18:06:31.280053 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:31.280058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:31.280178 (+ 120us) server_negotiation.cc:407] Connection header received
0923 18:06:31.280323 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:31.280327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:31.280390 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:31.280495 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:31.281259 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.282052 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.283038 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.283319 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.283437 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.286805 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.195:43429: BlockingRecv error: recv got EOF from 127.18.44.195:43429 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:31 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.193@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:31.778857 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:31.771578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.193:33781)
0923 18:06:31.771928 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:31.771943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:31.772064 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:31.772387 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:31.772390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:31.772701 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:31.772837 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.772844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.773774 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.773778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.774461 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:31.774471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.775112 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.775116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:31.775198 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.775962 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:31.775975 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:31.778625 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":133,"threads_started":1}
W20260923 18:06:31.779114 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:31.771802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:33781 (local address 127.18.44.254:36411)
0923 18:06:31.771959 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:31.771963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:31.772014 (+ 51us) server_negotiation.cc:407] Connection header received
0923 18:06:31.772195 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:31.772197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:31.772253 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:31.772334 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:31.772990 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.773552 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.774694 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:31.774954 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:31.775057 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:31.778915 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.193:33781: BlockingRecv error: recv got EOF from 127.18.44.193:33781 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 18:06:32 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:06:32 dist-test-slave-rvjj krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.44.194@KRBTEST.COM for oryx/127.18.44.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:06:32.043121 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.035642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36411 (local address 127.18.44.194:40735)
0923 18:06:32.036006 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:32.036024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:32.036175 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:32.036730 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:32.036734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:32.036984 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:32.037124 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.037132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.038322 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.038327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.039037 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.039053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.039653 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.039657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.039739 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.040351 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:32.040364 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:32.042950 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.44.254:36411: Server oryx/127.18.44.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
W20260923 18:06:32.043385 27799 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:32.035789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:40735 (local address 127.18.44.254:36411)
0923 18:06:32.036170 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:32.036178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:32.036203 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:06:32.036257 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:32.036260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:32.036312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:32.036503 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:32.037351 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.038102 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.039198 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.039520 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.039633 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.043145 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.44.194:40735: BlockingRecv error: recv got EOF from 127.18.44.194:40735 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 18:06:32.129330 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27372
I20260923 18:06:32.136107 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27524
I20260923 18:06:32.143501 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27656
I20260923 18:06:32.149497 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27302
2026-09-23T18:06:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18439 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T18:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:32Z Disabled control of system clock
I20260923 18:06:32.184469 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:43083
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:43083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 18:06:32.306550 27840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:32.306846 27840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:32.306881 27840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:32.311095 27840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:32.311203 27840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:32.311216 27840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:32.311232 27840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:32.311247 27840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:32.316399 27840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:43083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:43083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:32.317519 27840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:32.318892 27840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:06:32.329181 27840 server_base.cc:1034] running on GCE node
W20260923 18:06:32.329262 27848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:32.329388 27846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:32.329265 27845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:32.330368 27840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:32.332446 27840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:32.333858 27840 hybrid_clock.cc:648] HybridClock initialized: now 1790186792333807 us; error 119 us; skew 500 ppm
I20260923 18:06:32.336822 27840 webserver.cc:460] Webserver started at http://127.18.44.254:38909/ using document root <none> and password file <none>
I20260923 18:06:32.337661 27840 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:32.337750 27840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:32.338025 27840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:32.341015 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "8ba9e52935a04b56ad2a72d743131898"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "08a79bc5a4d41f1d5f5b6e69185498ef"
server_key_iv: "2ef36679431dc62085436e76b626c659"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.341833 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "8ba9e52935a04b56ad2a72d743131898"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "08a79bc5a4d41f1d5f5b6e69185498ef"
server_key_iv: "2ef36679431dc62085436e76b626c659"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.347683 27840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:06:32.351632 27854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.353770 27840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:32.354028 27840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "8ba9e52935a04b56ad2a72d743131898"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "08a79bc5a4d41f1d5f5b6e69185498ef"
server_key_iv: "2ef36679431dc62085436e76b626c659"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.354238 27840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:32.375949 27840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:32.377106 27840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:32.377372 27840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:32.386880 27840 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:43083
I20260923 18:06:32.386880 27906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:43083 every 8 connection(s)
I20260923 18:06:32.388353 27840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:32.392385 27907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:32.393683 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27840
I20260923 18:06:32.393808 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:32.394160 18611 external_mini_cluster.cc:1428] Setting key 228db1ef8efe353775714443327eb2c5
I20260923 18:06:32.400627 27907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898: Bootstrap starting.
I20260923 18:06:32.403788 27907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:32.404774 27907 log.cc:824] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:32.407631 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.396310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.18.44.254:43083)
0923 18:06:32.397202 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:32.397224 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:32.397276 (+ 52us) server_negotiation.cc:407] Connection header received
0923 18:06:32.398210 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:32.398234 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:32.398812 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:32.399240 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:32.399802 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.401014 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.402474 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.403167 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.403477 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.404592 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:32.404599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:32.404620 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:32.405273 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:32.405380 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:32.405410 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:32.405530 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:32.405826 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 18:06:32.406275 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":184,"threads_started":1}
I20260923 18:06:32.408161 27907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898: No bootstrap required, opened a new log
I20260923 18:06:32.413826 27907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } }
I20260923 18:06:32.414258 27907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:32.414319 27907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba9e52935a04b56ad2a72d743131898, State: Initialized, Role: FOLLOWER
I20260923 18:06:32.414958 27907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } }
I20260923 18:06:32.415165 27907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:32.415246 27907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:32.415412 27907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:32.417054 27907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } }
I20260923 18:06:32.417798 27907 leader_election.cc:302] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ba9e52935a04b56ad2a72d743131898; no voters:
I20260923 18:06:32.418316 27907 leader_election.cc:288] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:32.418567 27912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:32.418952 27912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba9e52935a04b56ad2a72d743131898, State: Running, Role: LEADER
I20260923 18:06:32.419452 27912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } }
I20260923 18:06:32.420207 27907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:32.421715 27914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ba9e52935a04b56ad2a72d743131898. Latest consensus state: current_term: 1 leader_uuid: "8ba9e52935a04b56ad2a72d743131898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } } }
I20260923 18:06:32.421765 27913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ba9e52935a04b56ad2a72d743131898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba9e52935a04b56ad2a72d743131898" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 43083 } } }
I20260923 18:06:32.421937 27914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:32.421945 27913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:32.422662 27921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:32.428150 27921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:32.437392 27921 catalog_manager.cc:1347] Generated new cluster ID: aac905e53df34ee0b178d7443d36aca4
I20260923 18:06:32.437569 27921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:32.449880 27921 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:32.451229 27921 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:32.457235 27921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ba9e52935a04b56ad2a72d743131898: Generated new TSK 0
I20260923 18:06:32.458312 27921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:06:32.467638 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--builtin_ntp_servers=127.18.44.212:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 18:06:32.614271 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:32.614717 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:32.614766 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:32.620046 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:32.620157 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:32.620227 27931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:32.627526 27931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:32.629653 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:32.632035 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:32.645015 27939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:32.645233 27937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:32.645604 27931 server_base.cc:1034] running on GCE node
W20260923 18:06:32.645483 27936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:32.646281 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:32.647075 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:32.648283 27931 hybrid_clock.cc:648] HybridClock initialized: now 1790186792648263 us; error 39 us; skew 500 ppm
I20260923 18:06:32.650717 27931 webserver.cc:460] Webserver started at http://127.18.44.193:35691/ using document root <none> and password file <none>
I20260923 18:06:32.651448 27931 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:32.651530 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:32.651772 27931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:32.654100 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "b066dfd692ac494c9144aa6529fc0201"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "7926d609e1d9dbd27e5ceea1d45fdecd"
server_key_iv: "58df3fd37a52be929cae2c5adc977c5e"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.654723 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "b066dfd692ac494c9144aa6529fc0201"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "7926d609e1d9dbd27e5ceea1d45fdecd"
server_key_iv: "58df3fd37a52be929cae2c5adc977c5e"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.659602 27931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 18:06:32.663178 27945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.665061 27931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:32.665220 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "b066dfd692ac494c9144aa6529fc0201"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "7926d609e1d9dbd27e5ceea1d45fdecd"
server_key_iv: "58df3fd37a52be929cae2c5adc977c5e"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.665377 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:32.684495 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:32.685313 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:32.685501 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:32.686157 27931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:32.687305 27931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:32.687376 27931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.687428 27931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:32.687463 27931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.698099 27931 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:42875
I20260923 18:06:32.698179 28058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:42875 every 8 connection(s)
I20260923 18:06:32.699193 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:32.707067 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 27931
I20260923 18:06:32.707331 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:32.707890 18611 external_mini_cluster.cc:1428] Setting key 530cfc23cbf3f1f85476c48bfe75f4e7
I20260923 18:06:32.711001 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--builtin_ntp_servers=127.18.44.212:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 18:06:32.712740 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.701709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:45983 (local address 127.18.44.254:43083)
0923 18:06:32.701950 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:32.701956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:32.703476 (+ 1520us) server_negotiation.cc:407] Connection header received
0923 18:06:32.705246 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:32.705249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:32.705318 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:32.705440 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:32.706800 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.707602 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.709611 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.709944 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.710062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.712156 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:32.712160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:32.712162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:32.712177 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:32.712239 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:32.712248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:32.712317 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:32.712532 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 18:06:32.712607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 18:06:32.714179 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.702202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:43083 (local address 127.18.44.193:45983)
0923 18:06:32.703197 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:32.703306 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:32.704894 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:32.705781 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:32.705792 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:32.705918 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:32.706601 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.706618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.707829 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.707836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.709397 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.709417 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.710205 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.710211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.710597 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.711651 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:32.711654 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:32.711755 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:32.711891 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:32.712344 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:32.712355 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:32.712363 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:06:32.712537 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:06:32.713052 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":139,"threads_started":1}
I20260923 18:06:32.716439 28059 heartbeater.cc:347] Connected to a master server at 127.18.44.254:43083
I20260923 18:06:32.716871 28059 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:32.717712 28059 heartbeater.cc:511] Master 127.18.44.254:43083 requested a full tablet report, sending...
I20260923 18:06:32.720230 27871 ts_manager.cc:194] Registered new tserver with Master: b066dfd692ac494c9144aa6529fc0201 (127.18.44.193:42875)
I20260923 18:06:32.722028 27871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.193:45983
W20260923 18:06:32.840615 28062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:32.840991 28062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:32.841035 28062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:32.845520 28062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:32.845608 28062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:32.845671 28062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:32.850787 28062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:32.852031 28062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:32.853535 28062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:32.864470 28068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:32.864460 28070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:32.864599 28062 server_base.cc:1034] running on GCE node
W20260923 18:06:32.864460 28067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:32.865319 28062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:32.866031 28062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:32.867244 28062 hybrid_clock.cc:648] HybridClock initialized: now 1790186792867221 us; error 63 us; skew 500 ppm
I20260923 18:06:32.869544 28062 webserver.cc:460] Webserver started at http://127.18.44.194:46631/ using document root <none> and password file <none>
I20260923 18:06:32.870260 28062 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:32.870335 28062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:32.870536 28062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:32.872730 28062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "33fa4597d7c746838eb8c64dea3b17ab"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "cb49ce161efd308a48c033808496df89"
server_key_iv: "a309eed280f9f93b4932fa6d66fe79d6"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.873312 28062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "33fa4597d7c746838eb8c64dea3b17ab"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "cb49ce161efd308a48c033808496df89"
server_key_iv: "a309eed280f9f93b4932fa6d66fe79d6"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.877789 28062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:06:32.881047 28076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.882814 28062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:06:32.882992 28062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "33fa4597d7c746838eb8c64dea3b17ab"
format_stamp: "Formatted at 2026-09-23 18:06:32 on dist-test-slave-rvjj"
server_key: "cb49ce161efd308a48c033808496df89"
server_key_iv: "a309eed280f9f93b4932fa6d66fe79d6"
server_key_version: "encryptionkey@0"
I20260923 18:06:32.883234 28062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:32.908459 28062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:32.909190 28062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:32.909371 28062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:32.909964 28062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:32.910988 28062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:32.911047 28062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.911105 28062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:32.911134 28062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:32.921406 28062 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:43605
I20260923 18:06:32.921521 28189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:43605 every 8 connection(s)
I20260923 18:06:32.922520 28062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:32.931314 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28062
I20260923 18:06:32.931473 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:32.931797 18611 external_mini_cluster.cc:1428] Setting key e163e43c34d71aa062ea19aaaebcf5a3
I20260923 18:06:32.933821 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.925130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:53915 (local address 127.18.44.254:43083)
0923 18:06:32.925261 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:32.925265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:32.927025 (+ 1760us) server_negotiation.cc:407] Connection header received
0923 18:06:32.928573 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:32.928576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:32.928643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:32.928721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:32.929731 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.930274 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.931606 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.931846 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.931918 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.933214 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:32.933217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:32.933220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:32.933237 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:32.933257 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:32.933265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:32.933326 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:32.933545 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 18:06:32.933646 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:06:32.934454 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--builtin_ntp_servers=127.18.44.212:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 18:06:32.934780 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:32.925652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:43083 (local address 127.18.44.194:53915)
0923 18:06:32.926806 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:32.926864 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:32.928313 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:32.928929 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:32.928937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:32.929044 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:32.929573 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.929587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.930398 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.930401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.931457 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:32.931468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:32.932099 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:32.932102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:32.932277 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:32.932924 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:32.932926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:32.932971 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:32.933046 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:32.933342 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:32.933351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:32.933357 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:06:32.933513 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 18:06:32.933828 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":207,"threads_started":1}
I20260923 18:06:32.936378 28190 heartbeater.cc:347] Connected to a master server at 127.18.44.254:43083
I20260923 18:06:32.936659 28190 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:32.937242 28190 heartbeater.cc:511] Master 127.18.44.254:43083 requested a full tablet report, sending...
I20260923 18:06:32.938589 27871 ts_manager.cc:194] Registered new tserver with Master: 33fa4597d7c746838eb8c64dea3b17ab (127.18.44.194:43605)
I20260923 18:06:32.939239 27871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.194:53915
W20260923 18:06:33.095068 28193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:33.095642 28193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:33.095716 28193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:33.100773 28193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:33.100914 28193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:33.101097 28193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:33.108489 28193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:43083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:33.110143 28193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:33.112154 28193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:33.125793 28201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.125793 28198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.126709 28199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:33.126890 28193 server_base.cc:1034] running on GCE node
I20260923 18:06:33.127480 28193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:33.128275 28193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:33.129631 28193 hybrid_clock.cc:648] HybridClock initialized: now 1790186793129523 us; error 115 us; skew 500 ppm
I20260923 18:06:33.132233 28193 webserver.cc:460] Webserver started at http://127.18.44.195:39787/ using document root <none> and password file <none>
I20260923 18:06:33.132913 28193 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:33.133011 28193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:33.133251 28193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:33.135911 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "51d6b7cc24af468b8b5a53ea5a1f1d6e"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "a886ed58c61cc6b19c27101df95f3172"
server_key_iv: "26ad375ead050d31fb8d285157415854"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.136554 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "51d6b7cc24af468b8b5a53ea5a1f1d6e"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "a886ed58c61cc6b19c27101df95f3172"
server_key_iv: "26ad375ead050d31fb8d285157415854"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.141218 28193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:33.144963 28207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.146765 28193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:06:33.146934 28193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "51d6b7cc24af468b8b5a53ea5a1f1d6e"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "a886ed58c61cc6b19c27101df95f3172"
server_key_iv: "26ad375ead050d31fb8d285157415854"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.147082 28193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:33.174001 28193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:33.174795 28193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:33.174968 28193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:33.175696 28193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:33.176779 28193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:33.176841 28193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.176889 28193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:33.176904 28193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.188773 28193 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:38139
I20260923 18:06:33.188823 28320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:38139 every 8 connection(s)
I20260923 18:06:33.189826 28193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:33.194406 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28193
I20260923 18:06:33.194571 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:33.194921 18611 external_mini_cluster.cc:1428] Setting key 82acc772ec36ec9bb60d3a37d3751b58
I20260923 18:06:33.203006 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.192558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:47547 (local address 127.18.44.254:43083)
0923 18:06:33.192778 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:33.192783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:33.194298 (+ 1515us) server_negotiation.cc:407] Connection header received
0923 18:06:33.196324 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:33.196327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:33.196386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:33.196454 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:33.197901 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:33.198495 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:33.200123 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:33.200406 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:33.200525 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:33.202507 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:33.202511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:33.202513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:33.202530 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:33.202560 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:33.202569 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:33.202647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:33.202796 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 18:06:33.202868 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:06:33.204893 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.192937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:43083 (local address 127.18.44.195:47547)
0923 18:06:33.194002 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:33.194088 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:33.196012 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:33.196713 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:33.196726 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:33.196884 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:33.197726 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:33.197746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:33.198670 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:33.198673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:33.199961 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:33.199976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:33.200641 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:33.200645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:33.201067 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:33.202074 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:33.202077 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:33.202160 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:33.202274 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:33.202662 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:33.202669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:33.202674 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:06:33.202807 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 18:06:33.203324 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":129,"threads_started":1}
I20260923 18:06:33.206745 28321 heartbeater.cc:347] Connected to a master server at 127.18.44.254:43083
I20260923 18:06:33.207126 28321 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:33.208056 28321 heartbeater.cc:511] Master 127.18.44.254:43083 requested a full tablet report, sending...
I20260923 18:06:33.209414 27871 ts_manager.cc:194] Registered new tserver with Master: 51d6b7cc24af468b8b5a53ea5a1f1d6e (127.18.44.195:38139)
I20260923 18:06:33.210106 27871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.195:47547
I20260923 18:06:33.210596 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:06:33.214401 27910 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:33.212962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.18.44.254:43083)
0923 18:06:33.213099 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:33.213103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:33.213444 (+ 341us) server_negotiation.cc:407] Connection header received
0923 18:06:33.213533 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:33.213536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:33.213650 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:33.213754 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:33.214253 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55168: BlockingRecv error: recv got EOF from 127.0.0.1:55168 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:06:33.215842 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27931
I20260923 18:06:33.222924 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28062
I20260923 18:06:33.229787 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28193
I20260923 18:06:33.235607 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 27840
2026-09-23T18:06:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1084 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T18:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:33Z Disabled control of system clock
I20260923 18:06:33.271314 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:35899
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:35899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 18:06:33.441241 28332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:33.441787 28332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:33.441859 28332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:33.449118 28332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:33.449477 28332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:33.449574 28332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:33.449599 28332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:33.449616 28332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:33.459456 28332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:35899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:35899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:33.461164 28332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:33.463172 28332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:33.477011 28340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.477049 28338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:33.477272 28332 server_base.cc:1034] running on GCE node
W20260923 18:06:33.477336 28337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:33.478113 28332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:33.479593 28332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:33.480809 28332 hybrid_clock.cc:648] HybridClock initialized: now 1790186793480796 us; error 64 us; skew 500 ppm
I20260923 18:06:33.483059 28332 webserver.cc:460] Webserver started at http://127.18.44.254:43867/ using document root <none> and password file <none>
I20260923 18:06:33.483768 28332 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:33.483845 28332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:33.484026 28332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:33.486279 28332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "4b9419def235421aba509e52ac4f72b7"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "d56e6ac2cc16eae4ebf853aa582fdd9f"
server_key_iv: "c00a7e598de8a369848dd3c66c6cfb4e"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.486858 28332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "4b9419def235421aba509e52ac4f72b7"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "d56e6ac2cc16eae4ebf853aa582fdd9f"
server_key_iv: "c00a7e598de8a369848dd3c66c6cfb4e"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.491540 28332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:06:33.494818 28346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.496695 28332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:33.496876 28332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "4b9419def235421aba509e52ac4f72b7"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "d56e6ac2cc16eae4ebf853aa582fdd9f"
server_key_iv: "c00a7e598de8a369848dd3c66c6cfb4e"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.496997 28332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:33.502163 28332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:33.502885 28332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:33.503057 28332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:33.511165 28332 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:35899
I20260923 18:06:33.511171 28398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:35899 every 8 connection(s)
I20260923 18:06:33.512343 28332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:33.517124 28399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:33.522284 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28332
I20260923 18:06:33.522418 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:33.522729 18611 external_mini_cluster.cc:1428] Setting key ff4440e8e63cc0cec1d279807205f7b5
I20260923 18:06:33.525674 28399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7: Bootstrap starting.
I20260923 18:06:33.529546 28399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:33.530920 28399 log.cc:824] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:33.531168 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.524829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.18.44.254:35899)
0923 18:06:33.525732 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:33.525751 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:33.525801 (+ 50us) server_negotiation.cc:407] Connection header received
0923 18:06:33.526781 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:33.526807 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:33.527340 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:33.527884 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:33.529262 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:33.529268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:33.529285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:33.529301 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:33.529388 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:33.529408 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:33.529535 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:33.529860 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 18:06:33.530215 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":170,"threads_started":1}
I20260923 18:06:33.533294 28399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7: No bootstrap required, opened a new log
I20260923 18:06:33.538825 28399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } }
I20260923 18:06:33.539290 28399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:33.539368 28399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9419def235421aba509e52ac4f72b7, State: Initialized, Role: FOLLOWER
I20260923 18:06:33.540017 28399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } }
I20260923 18:06:33.540220 28399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:33.540304 28399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:33.540424 28399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:33.541766 28399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } }
I20260923 18:06:33.542254 28399 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9419def235421aba509e52ac4f72b7; no voters:
I20260923 18:06:33.542809 28399 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:33.543350 28404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:33.543864 28404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4b9419def235421aba509e52ac4f72b7, State: Running, Role: LEADER
I20260923 18:06:33.544492 28404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } }
I20260923 18:06:33.544986 28399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:33.546772 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9419def235421aba509e52ac4f72b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } } }
I20260923 18:06:33.546990 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:33.547235 28406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9419def235421aba509e52ac4f72b7. Latest consensus state: current_term: 1 leader_uuid: "4b9419def235421aba509e52ac4f72b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9419def235421aba509e52ac4f72b7" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 35899 } } }
I20260923 18:06:33.547444 28406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:33.547636 28413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:33.551688 28413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:33.560706 28413 catalog_manager.cc:1347] Generated new cluster ID: 725f1c4d6b8f4798a655a8ff0bf83a3d
I20260923 18:06:33.560855 28413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:33.576238 28413 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:33.576750 28413 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:33.584208 28413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9419def235421aba509e52ac4f72b7: Generated new TSK 0
I20260923 18:06:33.585373 28413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:06:33.594393 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--builtin_ntp_servers=127.18.44.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 18:06:33.736187 28423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:33.736498 28423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:33.736541 28423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:33.740891 28423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:33.740988 28423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:33.741050 28423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:33.746652 28423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:33.747920 28423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:33.749397 28423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:33.759816 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.759748 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.759791 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:33.760572 28423 server_base.cc:1034] running on GCE node
I20260923 18:06:33.761108 28423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:33.761801 28423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:33.762972 28423 hybrid_clock.cc:648] HybridClock initialized: now 1790186793762948 us; error 42 us; skew 500 ppm
I20260923 18:06:33.765239 28423 webserver.cc:460] Webserver started at http://127.18.44.193:42591/ using document root <none> and password file <none>
I20260923 18:06:33.765851 28423 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:33.765923 28423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:33.766156 28423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:33.768414 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "a6dcc89000ef4aeb9ee1ed2958050bf9"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "785fb14c95d8b55bf35ee2e26deb22a8"
server_key_iv: "51d10519f680c2e428b5b97c04105cd0"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.768999 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "a6dcc89000ef4aeb9ee1ed2958050bf9"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "785fb14c95d8b55bf35ee2e26deb22a8"
server_key_iv: "51d10519f680c2e428b5b97c04105cd0"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.773135 28423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:06:33.776271 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.777882 28423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:06:33.778021 28423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "a6dcc89000ef4aeb9ee1ed2958050bf9"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "785fb14c95d8b55bf35ee2e26deb22a8"
server_key_iv: "51d10519f680c2e428b5b97c04105cd0"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.778155 28423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:33.783183 28423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:33.783986 28423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:33.784183 28423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:33.784797 28423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:33.785830 28423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:33.785880 28423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.785936 28423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:33.785966 28423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.795087 28423 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:42119
I20260923 18:06:33.795128 28550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:42119 every 8 connection(s)
I20260923 18:06:33.796167 28423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:33.803042 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.798305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35147 (local address 127.18.44.254:35899)
0923 18:06:33.798446 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:33.798450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:33.799636 (+ 1186us) server_negotiation.cc:407] Connection header received
0923 18:06:33.801044 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:33.801046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:33.801111 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:33.801234 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:33.802464 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:33.802467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:33.802469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:33.802482 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:33.802505 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:33.802513 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:33.802608 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:33.802870 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 18:06:33.802917 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:06:33.803646 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28423
I20260923 18:06:33.803761 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:33.804055 18611 external_mini_cluster.cc:1428] Setting key 52759b66bff29f71d974c8c847c10882
I20260923 18:06:33.804338 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.798607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:35899 (local address 127.18.44.193:35147)
0923 18:06:33.799458 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:33.799508 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:33.800816 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:33.801427 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:33.801435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:33.801537 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:33.802145 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:33.802147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:33.802201 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:33.802288 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:33.802703 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:33.802713 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:33.802718 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:06:33.802883 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 18:06:33.803117 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":118,"threads_started":1}
I20260923 18:06:33.805748 28551 heartbeater.cc:347] Connected to a master server at 127.18.44.254:35899
I20260923 18:06:33.805997 28551 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:33.806270 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--builtin_ntp_servers=127.18.44.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 18:06:33.806463 28551 heartbeater.cc:511] Master 127.18.44.254:35899 requested a full tablet report, sending...
I20260923 18:06:33.808123 28363 ts_manager.cc:194] Registered new tserver with Master: a6dcc89000ef4aeb9ee1ed2958050bf9 (127.18.44.193:42119)
W20260923 18:06:33.921271 28554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:33.921595 28554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:33.921664 28554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:33.926335 28554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:33.926437 28554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:33.926524 28554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:33.931756 28554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:33.932894 28554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:33.934343 28554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:33.944836 28562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.944864 28560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:33.944952 28559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:33.945164 28554 server_base.cc:1034] running on GCE node
I20260923 18:06:33.945709 28554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:33.946441 28554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:33.947654 28554 hybrid_clock.cc:648] HybridClock initialized: now 1790186793947610 us; error 41 us; skew 500 ppm
I20260923 18:06:33.949882 28554 webserver.cc:460] Webserver started at http://127.18.44.194:37683/ using document root <none> and password file <none>
I20260923 18:06:33.950587 28554 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:33.950668 28554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:33.950889 28554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:33.953261 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "49158fd4b81545eb88d46e34c6d83705"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "494fd6827f5bd4c6149e71df5dda76b4"
server_key_iv: "6f1c2c45461b677b7c8a0607ea42725f"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.953848 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "49158fd4b81545eb88d46e34c6d83705"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "494fd6827f5bd4c6149e71df5dda76b4"
server_key_iv: "6f1c2c45461b677b7c8a0607ea42725f"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.957975 28554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:06:33.960960 28568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.962445 28554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:06:33.962567 28554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "49158fd4b81545eb88d46e34c6d83705"
format_stamp: "Formatted at 2026-09-23 18:06:33 on dist-test-slave-rvjj"
server_key: "494fd6827f5bd4c6149e71df5dda76b4"
server_key_iv: "6f1c2c45461b677b7c8a0607ea42725f"
server_key_version: "encryptionkey@0"
I20260923 18:06:33.962695 28554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:33.967473 28554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:33.968165 28554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:33.968339 28554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:33.968907 28554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:33.969890 28554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:33.969983 28554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.970042 28554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:33.970072 28554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:33.979552 28554 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:37423
I20260923 18:06:33.979583 28681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:37423 every 8 connection(s)
I20260923 18:06:33.980535 28554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:33.982898 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28554
I20260923 18:06:33.983017 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:33.983377 18611 external_mini_cluster.cc:1428] Setting key 6365fca85571feec3eb45bf577f05c9e
I20260923 18:06:33.985297 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--builtin_ntp_servers=127.18.44.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 18:06:33.987769 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.982595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:36665 (local address 127.18.44.254:35899)
0923 18:06:33.982836 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:33.982841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:33.984127 (+ 1286us) server_negotiation.cc:407] Connection header received
0923 18:06:33.985533 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:33.985536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:33.985581 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:33.985623 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:33.987113 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:33.987116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:33.987118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:33.987132 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:33.987159 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:33.987169 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:33.987302 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:33.987505 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 18:06:33.987561 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:06:33.988926 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:33.983095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:35899 (local address 127.18.44.194:36665)
0923 18:06:33.983966 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:33.984026 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:33.985300 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:33.985956 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:33.985964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:33.986059 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:33.986727 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:33.986730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:33.986793 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:33.986897 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:33.987295 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:33.987304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:33.987311 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:06:33.987523 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 18:06:33.987855 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":194,"threads_started":1}
I20260923 18:06:33.990736 28682 heartbeater.cc:347] Connected to a master server at 127.18.44.254:35899
I20260923 18:06:33.991086 28682 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:33.991657 28682 heartbeater.cc:511] Master 127.18.44.254:35899 requested a full tablet report, sending...
I20260923 18:06:33.993110 28363 ts_manager.cc:194] Registered new tserver with Master: 49158fd4b81545eb88d46e34c6d83705 (127.18.44.194:37423)
W20260923 18:06:34.109818 28685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:34.110116 28685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:34.110162 28685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:34.114496 28685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:34.114583 28685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:34.114645 28685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:34.119652 28685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:34.120832 28685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:34.122208 28685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:34.132551 28691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:34.132640 28685 server_base.cc:1034] running on GCE node
W20260923 18:06:34.132596 28690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:34.132572 28693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:34.133407 28685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:34.134167 28685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:34.135349 28685 hybrid_clock.cc:648] HybridClock initialized: now 1790186794135323 us; error 46 us; skew 500 ppm
I20260923 18:06:34.137574 28685 webserver.cc:460] Webserver started at http://127.18.44.195:42157/ using document root <none> and password file <none>
I20260923 18:06:34.138211 28685 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:34.138285 28685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:34.138514 28685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:34.140949 28685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "58d1e8fd32ca4502a146169085cfe9cf"
format_stamp: "Formatted at 2026-09-23 18:06:34 on dist-test-slave-rvjj"
server_key: "4c62b7055eb4bf2e12c0212254f68732"
server_key_iv: "36af252c495451955f14920b884ab18d"
server_key_version: "encryptionkey@0"
I20260923 18:06:34.141558 28685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "58d1e8fd32ca4502a146169085cfe9cf"
format_stamp: "Formatted at 2026-09-23 18:06:34 on dist-test-slave-rvjj"
server_key: "4c62b7055eb4bf2e12c0212254f68732"
server_key_iv: "36af252c495451955f14920b884ab18d"
server_key_version: "encryptionkey@0"
I20260923 18:06:34.145761 28685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:06:34.148911 28699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:34.150434 28685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:06:34.150578 28685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "58d1e8fd32ca4502a146169085cfe9cf"
format_stamp: "Formatted at 2026-09-23 18:06:34 on dist-test-slave-rvjj"
server_key: "4c62b7055eb4bf2e12c0212254f68732"
server_key_iv: "36af252c495451955f14920b884ab18d"
server_key_version: "encryptionkey@0"
I20260923 18:06:34.150722 28685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:34.156100 28685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:34.156812 28685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:34.156967 28685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:34.157529 28685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:34.158500 28685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:34.158548 28685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:34.158596 28685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:34.158617 28685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:34.168462 28685 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:43803
I20260923 18:06:34.168479 28812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:43803 every 8 connection(s)
I20260923 18:06:34.169576 28685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:34.172886 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28685
I20260923 18:06:34.173000 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:34.173295 18611 external_mini_cluster.cc:1428] Setting key 66489d2f749e950438ea0b087edcad18
I20260923 18:06:34.176081 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:34.171555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:49939 (local address 127.18.44.254:35899)
0923 18:06:34.171771 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:34.171775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:34.172909 (+ 1134us) server_negotiation.cc:407] Connection header received
0923 18:06:34.174307 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:34.174310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:34.174357 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:34.174433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:34.175629 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:34.175633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:34.175635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:06:34.175649 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:34.175672 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:06:34.175679 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:34.175754 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:34.175902 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 18:06:34.175947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260923 18:06:34.176997 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:34.171946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:35899 (local address 127.18.44.195:49939)
0923 18:06:34.172728 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:34.172775 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:34.174072 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:34.174644 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:34.174652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:34.174755 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:34.175355 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:06:34.175356 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:34.175399 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:06:34.175476 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:34.175758 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:34.175768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:34.175774 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:06:34.175941 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 18:06:34.176211 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":119,"threads_started":1}
I20260923 18:06:34.178156 28813 heartbeater.cc:347] Connected to a master server at 127.18.44.254:35899
I20260923 18:06:34.178382 28813 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:34.178857 28813 heartbeater.cc:511] Master 127.18.44.254:35899 requested a full tablet report, sending...
I20260923 18:06:34.180033 28363 ts_manager.cc:194] Registered new tserver with Master: 58d1e8fd32ca4502a146169085cfe9cf (127.18.44.195:43803)
I20260923 18:06:34.187107 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:06:34.190274 28402 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:06:34.189384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.18.44.254:35899)
0923 18:06:34.189529 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:34.189532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:34.189655 (+ 123us) server_negotiation.cc:407] Connection header received
0923 18:06:34.189712 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:34.189720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:34.189759 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:34.189846 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:34.190144 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60054: BlockingRecv error: recv got EOF from 127.0.0.1:60054 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:06:34.190903 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28423
I20260923 18:06:34.196892 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28554
I20260923 18:06:34.202194 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28685
I20260923 18:06:34.207332 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28332
2026-09-23T18:06:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (969 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:34 dist-test-slave-rvjj krb5kdc[28823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:34 dist-test-slave-rvjj krb5kdc[28823](info): set up 2 sockets
Sep 23 18:06:34 dist-test-slave-rvjj krb5kdc[28823](info): commencing operation
krb5kdc: starting...
W20260923 18:06:35.233389 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:35 dist-test-slave-rvjj krb5kdc[28823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:06:35.387802 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45881
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:35.504992 28836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:35.505330 28836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:35.505379 28836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:35.509778 28836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:06:35.509877 28836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:35.509896 28836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:35.509913 28836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:06:35.509927 28836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:06:35.515735 28836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:35.517151 28836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:35.518326 28836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:35.529960 28841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:35.529960 28844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:35.530169 28836 server_base.cc:1034] running on GCE node
W20260923 18:06:35.530578 28842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:35.531793 28836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:35.533576 28836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:35.537029 28836 hybrid_clock.cc:648] HybridClock initialized: now 1790186795536908 us; error 113 us; skew 500 ppm
Sep 23 18:06:35 dist-test-slave-rvjj krb5kdc[28823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:35.549352 28836 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:06:35.551043 28836 webserver.cc:460] Webserver started at http://127.18.44.254:46127/ using document root <none> and password file <none>
I20260923 18:06:35.552011 28836 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:35.552127 28836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:35.552392 28836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:35.556250 28836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "74a4ed2ce8274c98a4e02d67629d5da3"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "31df222b47a7cebfcf3f62c20da532ea"
server_key_iv: "8e2d6b31c7e54c87e579c6a868d137cb"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.557149 28836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "74a4ed2ce8274c98a4e02d67629d5da3"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "31df222b47a7cebfcf3f62c20da532ea"
server_key_iv: "8e2d6b31c7e54c87e579c6a868d137cb"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.564255 28836 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 18:06:35.568490 28851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:35.570647 28836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:06:35.570884 28836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "74a4ed2ce8274c98a4e02d67629d5da3"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "31df222b47a7cebfcf3f62c20da532ea"
server_key_iv: "8e2d6b31c7e54c87e579c6a868d137cb"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.571055 28836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:35.592818 28836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:35.597733 28836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:35.597991 28836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:35.607869 28836 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45881
I20260923 18:06:35.608178 28903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45881 every 8 connection(s)
I20260923 18:06:35.609305 28836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:06:35.612895 28904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:35.618114 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28836
I20260923 18:06:35.618291 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:06:35.618599 18611 external_mini_cluster.cc:1428] Setting key 1bf508016d8de495e51548e8278f18c0
I20260923 18:06:35.620152 28904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: Bootstrap starting.
I20260923 18:06:35.623346 28904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:35.624294 28904 log.cc:824] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:35.627540 28904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: No bootstrap required, opened a new log
I20260923 18:06:35.632588 28904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } }
I20260923 18:06:35.632941 28904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:35.632993 28904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4ed2ce8274c98a4e02d67629d5da3, State: Initialized, Role: FOLLOWER
I20260923 18:06:35.633550 28904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } }
I20260923 18:06:35.633759 28904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:06:35.633831 28904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:06:35.633951 28904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:35.635027 28904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } }
I20260923 18:06:35.635460 28904 leader_election.cc:302] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a4ed2ce8274c98a4e02d67629d5da3; no voters:
Sep 23 18:06:35 dist-test-slave-rvjj krb5kdc[28823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:35.635845 28904 leader_election.cc:288] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:06:35.636162 28909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:35.636626 28909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4ed2ce8274c98a4e02d67629d5da3, State: Running, Role: LEADER
I20260923 18:06:35.637324 28904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:06:35.637677 28909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } }
I20260923 18:06:35.641076 28910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } } }
I20260923 18:06:35.641443 28910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:35.642549 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74a4ed2ce8274c98a4e02d67629d5da3. Latest consensus state: current_term: 1 leader_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4ed2ce8274c98a4e02d67629d5da3" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45881 } } }
I20260923 18:06:35.642927 28923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:06:35.642988 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:06:35.646714 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:35.623807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.18.44.254:45881)
0923 18:06:35.624502 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:35.624524 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:35.624581 (+ 57us) server_negotiation.cc:407] Connection header received
0923 18:06:35.625446 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:35.625475 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:35.626069 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:35.626567 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:35.627773 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:35.630625 (+ 2852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:35.632058 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:35.632801 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:35.633035 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:35.636682 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:35.636696 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:35.636732 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:35.636975 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:35.640188 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:35.641052 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:35.641063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:35.641073 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:35.641627 (+ 554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:35.642191 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:35.642196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:35.642198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:35.643666 (+ 1468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:35.644000 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:35.644585 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 18:06:35.645544 (+ 959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":194,"threads_started":1}
I20260923 18:06:35.647583 28923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:06:35.657445 28923 catalog_manager.cc:1347] Generated new cluster ID: e80fce9fe6844b7192e1c8539d2c060e
I20260923 18:06:35.657577 28923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:06:35.676337 28923 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:06:35.677925 28923 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:06:35.688408 28923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: Generated new TSK 0
I20260923 18:06:35.689419 28923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:06:35.742050 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:35.899453 28930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:35.899899 28930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:35.899957 28930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:35.905275 28930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:35.905411 28930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:35.905496 28930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:06:35.912678 28930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:35.914253 28930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:35.915652 28930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:35.929731 28936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:35.929860 28938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:35.929858 28935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:35.930343 28930 server_base.cc:1034] running on GCE node
I20260923 18:06:35.930981 28930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:35.932051 28930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:35.933312 28930 hybrid_clock.cc:648] HybridClock initialized: now 1790186795933287 us; error 46 us; skew 500 ppm
Sep 23 18:06:35 dist-test-slave-rvjj krb5kdc[28823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:35.937415 28930 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:06:35.938833 28930 webserver.cc:460] Webserver started at http://127.18.44.193:41177/ using document root <none> and password file <none>
I20260923 18:06:35.939703 28930 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:35.939795 28930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:35.940062 28930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:35.942883 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "d5f81fb67c71452086793f979e46d101"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "09bff1334cc8d751b968c511f426a9a7"
server_key_iv: "3688aae453e0d66d30acb98ab696e9de"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.943739 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f81fb67c71452086793f979e46d101"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "09bff1334cc8d751b968c511f426a9a7"
server_key_iv: "3688aae453e0d66d30acb98ab696e9de"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.950773 28930 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260923 18:06:35.955091 28945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:35.957161 28930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:35.957407 28930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "d5f81fb67c71452086793f979e46d101"
format_stamp: "Formatted at 2026-09-23 18:06:35 on dist-test-slave-rvjj"
server_key: "09bff1334cc8d751b968c511f426a9a7"
server_key_iv: "3688aae453e0d66d30acb98ab696e9de"
server_key_version: "encryptionkey@0"
I20260923 18:06:35.957613 28930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:35.972350 28930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:35.976956 28930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:35.977223 28930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:35.978060 28930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:35.979493 28930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:35.979571 28930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:35.979632 28930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:35.979666 28930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:35.993011 28930 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:34367
I20260923 18:06:35.993103 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:34367 every 8 connection(s)
I20260923 18:06:35.994230 28930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:06:36.003840 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 28930
I20260923 18:06:36.003994 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:06:36.004314 18611 external_mini_cluster.cc:1428] Setting key 2395db1966e2fd7b9342ef3bde0c838d
Sep 23 18:06:36 dist-test-slave-rvjj krb5kdc[28823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186795, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:36.012923 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:35.996804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35509 (local address 127.18.44.254:45881)
0923 18:06:35.997032 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:35.997036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:35.998306 (+ 1270us) server_negotiation.cc:407] Connection header received
0923 18:06:36.000021 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.000024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.000101 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.000179 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:36.001632 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.002195 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.003533 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.003778 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.003876 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.007318 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:36.007322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:36.007325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:36.007358 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:36.009223 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.010205 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.010209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.010210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.010324 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.010958 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.010962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.010964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.011191 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:36.011402 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.012591 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.012765 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 18:06:36.014678 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:35.997371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45881 (local address 127.18.44.193:35509)
0923 18:06:35.998086 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:35.998149 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:35.999746 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.000415 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.000423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.000870 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:36.001445 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.001459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.002332 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.002336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.003397 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.003410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.003953 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.003957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.004167 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.004972 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:36.004995 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:36.007024 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:36.009444 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.009464 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.009494 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.010017 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.010523 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.010526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.010529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.010802 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.011423 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:36.011434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:36.012137 (+ 703us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.012486 (+ 349us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.013066 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":178,"threads_started":1}
I20260923 18:06:36.016681 29059 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45881
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:06:36.016987 29059 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:36.017578 29059 heartbeater.cc:511] Master 127.18.44.254:45881 requested a full tablet report, sending...
I20260923 18:06:36.019923 28866 ts_manager.cc:194] Registered new tserver with Master: d5f81fb67c71452086793f979e46d101 (127.18.44.193:34367)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:06:36.021718 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:35509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:06:36.043794 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:36.166736 29064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:36.167045 29064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:36.167111 29064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:36.171341 29064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:36.171425 29064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:36.171511 29064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:06:36.177251 29064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:36.178460 29064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:36.179569 29064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:36.190820 29069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:36.190968 29064 server_base.cc:1034] running on GCE node
W20260923 18:06:36.190820 29072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:36.191351 29070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:36.192077 29064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:36.193086 29064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:36.194319 29064 hybrid_clock.cc:648] HybridClock initialized: now 1790186796194238 us; error 97 us; skew 500 ppm
Sep 23 18:06:36 dist-test-slave-rvjj krb5kdc[28823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186796, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:36.198170 29064 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:06:36.200214 29064 webserver.cc:460] Webserver started at http://127.18.44.194:34583/ using document root <none> and password file <none>
I20260923 18:06:36.201069 29064 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:36.201176 29064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:36.201450 29064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:36.204385 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "c6f7f2587bb6419b97e8f89b5d16b911"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "9c4a92471ba84d452cc2ee3533ed0b5b"
server_key_iv: "dfe92152d38fb8459b8264d5f9a9047d"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.205186 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "c6f7f2587bb6419b97e8f89b5d16b911"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "9c4a92471ba84d452cc2ee3533ed0b5b"
server_key_iv: "dfe92152d38fb8459b8264d5f9a9047d"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.210561 29064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:06:36.214423 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.216472 29064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 18:06:36.216698 29064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "c6f7f2587bb6419b97e8f89b5d16b911"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "9c4a92471ba84d452cc2ee3533ed0b5b"
server_key_iv: "dfe92152d38fb8459b8264d5f9a9047d"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.216863 29064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:36.245661 29064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:36.249933 29064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:36.250172 29064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:36.250952 29064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:36.252089 29064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:36.252147 29064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.252213 29064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:36.252242 29064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.264891 29064 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39745
I20260923 18:06:36.264937 29192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39745 every 8 connection(s)
I20260923 18:06:36.266518 29064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:06:36.273825 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 29064
I20260923 18:06:36.273963 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:06:36.274320 18611 external_mini_cluster.cc:1428] Setting key b660b86d3182676f06e8c41f19c72171
Sep 23 18:06:36 dist-test-slave-rvjj krb5kdc[28823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186796, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:06:36.286017 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.269221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:50613 (local address 127.18.44.254:45881)
0923 18:06:36.269394 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.269398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.270685 (+ 1287us) server_negotiation.cc:407] Connection header received
0923 18:06:36.272345 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.272350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.272451 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.272577 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:36.274374 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.275429 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.277157 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.277536 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.277685 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.281359 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:36.281363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:36.281366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:36.281488 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:36.283133 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.283898 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.283903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.283905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.283998 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.284397 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.284400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.284401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.284649 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:36.284756 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.285645 (+ 889us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.285812 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:06:36.287238 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.269723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45881 (local address 127.18.44.194:50613)
0923 18:06:36.270455 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.270513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.271987 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.272807 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.272816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.273308 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:36.274087 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.274114 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.275720 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.275724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.276975 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.276990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.277825 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.277829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.278007 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.278891 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:36.278915 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:36.281077 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:36.283364 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.283374 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.283395 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.283777 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.284094 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.284097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.284100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.284282 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.284916 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:36.284927 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:36.285295 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.285526 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.286053 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":194,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:06:36.289234 29193 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45881
I20260923 18:06:36.289506 29193 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:36.290068 29193 heartbeater.cc:511] Master 127.18.44.254:45881 requested a full tablet report, sending...
I20260923 18:06:36.291503 28866 ts_manager.cc:194] Registered new tserver with Master: c6f7f2587bb6419b97e8f89b5d16b911 (127.18.44.194:39745)
I20260923 18:06:36.292250 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:50613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:06:36.308530 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:06:36.453505 29198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:06:36.453931 29198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:06:36.453985 29198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:06:36.458757 29198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:06:36.458879 29198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:06:36.458951 29198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:06:36.465732 29198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:06:36.467648 29198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:06:36.469165 29198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:06:36.483065 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:36.483196 29198 server_base.cc:1034] running on GCE node
W20260923 18:06:36.483134 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:06:36.483078 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:06:36.484018 29198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:06:36.484920 29198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:06:36.486166 29198 hybrid_clock.cc:648] HybridClock initialized: now 1790186796486124 us; error 62 us; skew 500 ppm
Sep 23 18:06:36 dist-test-slave-rvjj krb5kdc[28823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186796, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:06:36.490291 29198 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:06:36.491977 29198 webserver.cc:460] Webserver started at http://127.18.44.195:39139/ using document root <none> and password file <none>
I20260923 18:06:36.492782 29198 fs_manager.cc:359] Metadata directory not provided
I20260923 18:06:36.492877 29198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:06:36.493130 29198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:06:36.496022 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "83622dedc9354302b2ad55a2dd255fc0"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "e02312bafcbc412afc429f77de088800"
server_key_iv: "c23dbde67941b4e3c052b8f22838013b"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.496773 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "83622dedc9354302b2ad55a2dd255fc0"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "e02312bafcbc412afc429f77de088800"
server_key_iv: "c23dbde67941b4e3c052b8f22838013b"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.502075 29198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:06:36.506443 29213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.508809 29198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:06:36.509135 29198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "83622dedc9354302b2ad55a2dd255fc0"
format_stamp: "Formatted at 2026-09-23 18:06:36 on dist-test-slave-rvjj"
server_key: "e02312bafcbc412afc429f77de088800"
server_key_iv: "c23dbde67941b4e3c052b8f22838013b"
server_key_version: "encryptionkey@0"
I20260923 18:06:36.509440 29198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:06:36.524201 29198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:06:36.528638 29198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:06:36.528892 29198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:06:36.529711 29198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:06:36.530854 29198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:06:36.530911 29198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.530982 29198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:06:36.531018 29198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:06:36.543622 29198 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:38139
I20260923 18:06:36.543660 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:38139 every 8 connection(s)
I20260923 18:06:36.544874 29198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:06:36.548470 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 29198
I20260923 18:06:36.548621 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:06:36.548944 18611 external_mini_cluster.cc:1428] Setting key ca093890d6966b00d668b55df422a22a
Sep 23 18:06:36 dist-test-slave-rvjj krb5kdc[28823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186796, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:06:36.565047 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.547469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:59009 (local address 127.18.44.254:45881)
0923 18:06:36.547637 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.547642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.549014 (+ 1372us) server_negotiation.cc:407] Connection header received
0923 18:06:36.551240 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.551244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.551452 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.551644 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:36.553547 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.554152 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.555666 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.555915 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.556022 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.559705 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:36.559711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:36.559713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:36.559756 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:36.561386 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.562463 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.562468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.562469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.562596 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.563182 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.563185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.563187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.563522 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:36.563627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.564666 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.564873 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:06:36.565927 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.547819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45881 (local address 127.18.44.195:59009)
0923 18:06:36.548780 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.548854 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.550813 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.551936 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.551949 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.552618 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:06:36.553340 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.553360 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.554301 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.554304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.555499 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.555515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.556109 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.556114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.556328 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.557082 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:06:36.557104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:06:36.559317 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:06:36.561670 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.561679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.561728 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.562203 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.562761 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:06:36.562764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:06:36.562766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:06:36.563045 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:06:36.563906 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:06:36.563914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:06:36.564315 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.564554 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.564976 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":171,"threads_started":1}
I20260923 18:06:36.567233 29327 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45881
I20260923 18:06:36.567651 29327 heartbeater.cc:464] Registering TS with master...
I20260923 18:06:36.568352 29327 heartbeater.cc:511] Master 127.18.44.254:45881 requested a full tablet report, sending...
I20260923 18:06:36.570056 28866 ts_manager.cc:194] Registered new tserver with Master: 83622dedc9354302b2ad55a2dd255fc0 (127.18.44.195:38139)
I20260923 18:06:36.570727 28866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:59009
I20260923 18:06:36.577684 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:06:36.589144 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.580181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.18.44.254:45881)
0923 18:06:36.580349 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.580353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.580521 (+ 168us) server_negotiation.cc:407] Connection header received
0923 18:06:36.580655 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.580658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.580711 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.580799 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:06:36.581706 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.582321 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.583304 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.583532 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.583619 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.585249 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:06:36.585252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:06:36.585254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:06:36.585287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:06:36.586964 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.587648 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.587650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.587652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.587760 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:06:36.588032 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:06:36.588034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:06:36.588035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:06:36.588333 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:06:36.588456 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.588827 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.588962 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:06:36.593581 28866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:06:36.597070 28866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:06:36.620281 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.610610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39745 (local address 127.0.0.1:54064)
0923 18:06:36.612867 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.612887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.613051 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.613918 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.613922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.613942 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:36.614100 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.614107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.616347 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.616352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.617771 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.617787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.619424 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.619429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.619609 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.619679 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.619807 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.620012 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":39,"thread_start_us":119,"threads_started":1}
I20260923 18:06:36.620621 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.611554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.18.44.193:34367)
0923 18:06:36.612010 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.612019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.612644 (+ 625us) server_negotiation.cc:407] Connection header received
0923 18:06:36.613044 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.613049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.613358 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.613492 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:36.614027 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.615118 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.617380 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.618873 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.618958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.619167 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.620021 (+ 854us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.620440 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":82,"threads_started":1}
I20260923 18:06:36.621232 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.610902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34367 (local address 127.0.0.1:56722)
0923 18:06:36.612478 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.612543 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.612799 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.613680 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.613682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.613708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:36.613881 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.613890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.615313 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.615316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.617053 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.617062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.619303 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.619307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.619825 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.619860 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.620885 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.621058 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":445,"thread_start_us":139,"threads_started":1}
I20260923 18:06:36.621351 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.612007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:38139 (local address 127.0.0.1:50114)
0923 18:06:36.612582 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.612593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.612799 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.613670 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.613676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.613813 (+ 137us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:36.613894 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.613900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.616945 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.616949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.618542 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.618559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.620855 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.620859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.620972 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.620997 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.621088 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.621210 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":101,"threads_started":1}
I20260923 18:06:36.621948 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.612341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.18.44.195:38139)
0923 18:06:36.613211 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.613224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.613249 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:06:36.613323 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.613328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.613488 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.614071 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:36.614237 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.616618 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.619604 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.620668 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.621485 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.621669 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.621733 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.621820 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":191,"threads_started":1}
I20260923 18:06:36.625061 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.610770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.18.44.194:39745)
0923 18:06:36.612037 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.612055 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.613204 (+ 1149us) server_negotiation.cc:407] Connection header received
0923 18:06:36.613316 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.613322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.613615 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.613840 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:36.614402 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.616126 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.617984 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.619165 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.621223 (+ 2058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.623019 (+ 1796us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.623097 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.624876 (+ 1779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":177,"threads_started":1}
I20260923 18:06:36.626600 28993 tablet_service.cc:1467] Processing CreateTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 (DEFAULT_TABLE table=test-table [id=08f4b6811b3e4d8fad015b64ef6737b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:36.627753 29261 tablet_service.cc:1467] Processing CreateTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 (DEFAULT_TABLE table=test-table [id=08f4b6811b3e4d8fad015b64ef6737b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:36.628885 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5439d315b024ec6a41a3faa6a0d1141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:36.628937 29127 tablet_service.cc:1467] Processing CreateTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 (DEFAULT_TABLE table=test-table [id=08f4b6811b3e4d8fad015b64ef6737b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:06:36.630223 29261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5439d315b024ec6a41a3faa6a0d1141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:36.630290 29127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5439d315b024ec6a41a3faa6a0d1141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:06:36.642122 29344 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Bootstrap starting.
I20260923 18:06:36.642836 29345 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Bootstrap starting.
I20260923 18:06:36.643961 29346 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Bootstrap starting.
I20260923 18:06:36.646450 29345 tablet_bootstrap.cc:654] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:36.647619 29346 tablet_bootstrap.cc:654] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:36.647631 29344 tablet_bootstrap.cc:654] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Neither blocks nor log segments found. Creating new log.
I20260923 18:06:36.648984 29345 log.cc:824] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:36.649927 29346 log.cc:824] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:36.649927 29344 log.cc:824] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Log is configured to *not* fsync() on all Append() calls
I20260923 18:06:36.652271 29345 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: No bootstrap required, opened a new log
I20260923 18:06:36.652648 29345 ts_tablet_manager.cc:1397] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 18:06:36.652757 29346 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: No bootstrap required, opened a new log
I20260923 18:06:36.653035 29346 ts_tablet_manager.cc:1397] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 18:06:36.654284 29344 tablet_bootstrap.cc:492] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: No bootstrap required, opened a new log
I20260923 18:06:36.654623 29344 ts_tablet_manager.cc:1397] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260923 18:06:36.659224 29346 raft_consensus.cc:348] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.659641 29346 raft_consensus.cc:374] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:36.659701 29346 raft_consensus.cc:729] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f81fb67c71452086793f979e46d101, State: Initialized, Role: FOLLOWER
I20260923 18:06:36.660218 29346 consensus_queue.cc:260] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.661347 29059 heartbeater.cc:503] Master 127.18.44.254:45881 was elected leader, sending a full tablet report...
I20260923 18:06:36.662010 29346 ts_tablet_manager.cc:1428] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Time spent starting tablet: real 0.009s user 0.001s sys 0.008s
I20260923 18:06:36.663149 29345 raft_consensus.cc:348] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.663877 29345 raft_consensus.cc:374] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:36.663981 29345 raft_consensus.cc:729] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83622dedc9354302b2ad55a2dd255fc0, State: Initialized, Role: FOLLOWER
I20260923 18:06:36.664419 29344 raft_consensus.cc:348] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.664788 29344 raft_consensus.cc:374] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:06:36.664719 29345 consensus_queue.cc:260] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.664870 29344 raft_consensus.cc:729] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f7f2587bb6419b97e8f89b5d16b911, State: Initialized, Role: FOLLOWER
I20260923 18:06:36.665413 29344 consensus_queue.cc:260] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.666582 29345 ts_tablet_manager.cc:1428] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 18:06:36.666767 29344 ts_tablet_manager.cc:1428] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260923 18:06:36.668295 29327 heartbeater.cc:503] Master 127.18.44.254:45881 was elected leader, sending a full tablet report...
I20260923 18:06:36.668306 29193 heartbeater.cc:503] Master 127.18.44.254:45881 was elected leader, sending a full tablet report...
W20260923 18:06:36.746096 29060 tablet.cc:2369] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:06:36.768405 29194 tablet.cc:2369] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:06:36.796478 29328 tablet.cc:2369] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:06:36.862106 29352 raft_consensus.cc:482] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:06:36.862403 29352 raft_consensus.cc:504] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.864034 29352 leader_election.cc:288] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f81fb67c71452086793f979e46d101 (127.18.44.193:34367), 83622dedc9354302b2ad55a2dd255fc0 (127.18.44.195:38139)
I20260923 18:06:36.870045 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.864518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:54919 (local address 127.18.44.193:34367)
0923 18:06:36.864673 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.864677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.864865 (+ 188us) server_negotiation.cc:407] Connection header received
0923 18:06:36.865150 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.865153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.865206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.865346 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:36.865772 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.866584 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.868550 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.869228 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.869392 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.869448 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.869817 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.869894 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:06:36.870045 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.864928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:56735 (local address 127.18.44.195:38139)
0923 18:06:36.865066 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.865070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.865089 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:06:36.865146 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.865149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.865202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.865309 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:06:36.865802 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.866562 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.868547 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.869228 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.869392 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.869448 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.869817 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.869894 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:06:36.870155 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.864430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:38139 (local address 127.18.44.194:56735)
0923 18:06:36.864833 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.864847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.864995 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.865448 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.865451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.865464 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:36.865618 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.865626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.866725 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.866730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.868360 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.868375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.869476 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.869479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.869650 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.869697 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.869791 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.869924 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":145,"spinlock_wait_cycles":16384,"thread_start_us":132,"threads_started":1}
I20260923 18:06:36.870153 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.864363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34367 (local address 127.18.44.194:54919)
0923 18:06:36.864723 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.864742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:06:36.864995 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:06:36.865371 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:06:36.865375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:06:36.865396 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:06:36.865623 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.865628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.866725 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.866730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.868360 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:06:36.868375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.869447 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.869452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:06:36.869645 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.869693 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:06:36.869791 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 18:06:36.869959 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":192,"spinlock_wait_cycles":30336,"thread_start_us":134,"threads_started":1}
I20260923 18:06:36.871075 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5439d315b024ec6a41a3faa6a0d1141" candidate_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f81fb67c71452086793f979e46d101" is_pre_election: true
I20260923 18:06:36.871119 29281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5439d315b024ec6a41a3faa6a0d1141" candidate_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83622dedc9354302b2ad55a2dd255fc0" is_pre_election: true
I20260923 18:06:36.871446 29013 raft_consensus.cc:2454] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f7f2587bb6419b97e8f89b5d16b911 in term 0.
I20260923 18:06:36.871552 29281 raft_consensus.cc:2454] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f7f2587bb6419b97e8f89b5d16b911 in term 0.
I20260923 18:06:36.872044 29080 leader_election.cc:302] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83622dedc9354302b2ad55a2dd255fc0, c6f7f2587bb6419b97e8f89b5d16b911, d5f81fb67c71452086793f979e46d101; no voters:
I20260923 18:06:36.872622 29352 raft_consensus.cc:2781] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:06:36.872704 29352 raft_consensus.cc:482] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:06:36.872735 29352 raft_consensus.cc:3037] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:36.873991 29352 raft_consensus.cc:504] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.874431 29352 leader_election.cc:288] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [CANDIDATE]: Term 1 election: Requested vote from peers d5f81fb67c71452086793f979e46d101 (127.18.44.193:34367), 83622dedc9354302b2ad55a2dd255fc0 (127.18.44.195:38139)
I20260923 18:06:36.874953 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5439d315b024ec6a41a3faa6a0d1141" candidate_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f81fb67c71452086793f979e46d101"
I20260923 18:06:36.875017 29281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5439d315b024ec6a41a3faa6a0d1141" candidate_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83622dedc9354302b2ad55a2dd255fc0"
I20260923 18:06:36.875115 29013 raft_consensus.cc:3037] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:36.875172 29281 raft_consensus.cc:3037] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:06:36.876516 29013 raft_consensus.cc:2454] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f7f2587bb6419b97e8f89b5d16b911 in term 1.
I20260923 18:06:36.876694 29281 raft_consensus.cc:2454] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f7f2587bb6419b97e8f89b5d16b911 in term 1.
I20260923 18:06:36.876921 29080 leader_election.cc:302] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6f7f2587bb6419b97e8f89b5d16b911, d5f81fb67c71452086793f979e46d101; no voters:
I20260923 18:06:36.877180 29352 raft_consensus.cc:2781] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:06:36.877461 29352 raft_consensus.cc:686] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 1 LEADER]: Becoming Leader. State: Replica: c6f7f2587bb6419b97e8f89b5d16b911, State: Running, Role: LEADER
I20260923 18:06:36.877856 29352 consensus_queue.cc:237] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } }
I20260923 18:06:36.881472 28868 catalog_manager.cc:5495] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f7f2587bb6419b97e8f89b5d16b911 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f7f2587bb6419b97e8f89b5d16b911" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 } health_report { overall_health: UNKNOWN } } }
I20260923 18:06:36.926383 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.922599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.18.44.194:39745)
0923 18:06:36.922781 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:06:36.922785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:06:36.922798 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:06:36.922879 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:06:36.922882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:06:36.922940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:06:36.923038 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:06:36.923347 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.923879 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.924782 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:06:36.924998 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:06:36.925089 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:06:36.925457 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:06:36.925962 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:06:36.926077 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 18:06:36.926155 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 18:06:36.926225 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:06:36.934870 29013 raft_consensus.cc:1261] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 1 FOLLOWER]: Refusing update from remote peer c6f7f2587bb6419b97e8f89b5d16b911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:36.934932 29281 raft_consensus.cc:1261] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 1 FOLLOWER]: Refusing update from remote peer c6f7f2587bb6419b97e8f89b5d16b911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:06:36.935860 29352 consensus_queue.cc:1035] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f81fb67c71452086793f979e46d101" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:36.936146 29355 consensus_queue.cc:1035] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83622dedc9354302b2ad55a2dd255fc0" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 38139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:06:36.947558 29359 mvcc.cc:205] Tried to move back new op lower bound from 7332605120233918464 to 7332605120014721024. Current Snapshot: MvccSnapshot[applied={T|T < 7332605120233918464}]
I20260923 18:06:36.950038 29360 mvcc.cc:205] Tried to move back new op lower bound from 7332605120233918464 to 7332605120014721024. Current Snapshot: MvccSnapshot[applied={T|T < 7332605120233918464}]
I20260923 18:06:36.958336 28866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:36.958614 28866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:06:36.962215 28866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74a4ed2ce8274c98a4e02d67629d5da3: Sending DeleteTablet for 3 replicas of tablet f5439d315b024ec6a41a3faa6a0d1141
I20260923 18:06:36.963624 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:56722
I20260923 18:06:36.963660 29261 tablet_service.cc:1514] Processing DeleteTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:50114
I20260923 18:06:36.964001 29127 tablet_service.cc:1514] Processing DeleteTablet for tablet f5439d315b024ec6a41a3faa6a0d1141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:06:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:54064
I20260923 18:06:36.964638 29369 tablet_replica.cc:330] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101: stopping tablet replica
I20260923 18:06:36.964638 29368 tablet_replica.cc:330] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: stopping tablet replica
I20260923 18:06:36.964879 29370 tablet_replica.cc:330] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: stopping tablet replica
I20260923 18:06:36.965131 29368 raft_consensus.cc:2229] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:36.965131 29369 raft_consensus.cc:2229] T f5439d315b024ec6a41a3faa6a0d1141 P d5f81fb67c71452086793f979e46d101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:06:36.965291 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28930
I20260923 18:06:36.965281 29370 raft_consensus.cc:2229] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:06:36.966382 29370 raft_consensus.cc:2258] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:36.967983 29368 raft_consensus.cc:2258] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:06:36.969501 29370 ts_tablet_manager.cc:1905] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:36.970691 29368 ts_tablet_manager.cc:1905] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:06:36.972570 29370 ts_tablet_manager.cc:1918] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:36.972702 29370 log.cc:1196] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/f5439d315b024ec6a41a3faa6a0d1141
I20260923 18:06:36.973029 29370 ts_tablet_manager.cc:1939] T f5439d315b024ec6a41a3faa6a0d1141 P c6f7f2587bb6419b97e8f89b5d16b911: Deleting consensus metadata
I20260923 18:06:36.974423 29368 ts_tablet_manager.cc:1918] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:06:36.974591 29368 log.cc:1196] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/f5439d315b024ec6a41a3faa6a0d1141
I20260923 18:06:36.974522 28855 catalog_manager.cc:4841] TS c6f7f2587bb6419b97e8f89b5d16b911 (127.18.44.194:39745): tablet f5439d315b024ec6a41a3faa6a0d1141 (table test-table [id=08f4b6811b3e4d8fad015b64ef6737b3]) successfully deleted
I20260923 18:06:36.975003 29368 ts_tablet_manager.cc:1939] T f5439d315b024ec6a41a3faa6a0d1141 P 83622dedc9354302b2ad55a2dd255fc0: Deleting consensus metadata
I20260923 18:06:36.975436 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 29064
W20260923 18:06:36.975394 28852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:34367 (error 108)
I20260923 18:06:36.977811 28855 catalog_manager.cc:4841] TS 83622dedc9354302b2ad55a2dd255fc0 (127.18.44.195:38139): tablet f5439d315b024ec6a41a3faa6a0d1141 (table test-table [id=08f4b6811b3e4d8fad015b64ef6737b3]) successfully deleted
I20260923 18:06:36.978173 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:06:36.977815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34367 (local address 127.0.0.1:56732)
0923 18:06:36.977975 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 18:06:36.978053 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 18:06:36.978370 28852 catalog_manager.cc:4584] TS d5f81fb67c71452086793f979e46d101 (127.18.44.193:34367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5439d315b024ec6a41a3faa6a0d1141: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34367: connect: Connection refused (error 111)
I20260923 18:06:36.982141 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 29198
I20260923 18:06:36.989053 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 28836
2026-09-23T18:06:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2786 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:06:37 dist-test-slave-rvjj krb5kdc[29374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:06:37 dist-test-slave-rvjj krb5kdc[29374](info): set up 2 sockets
Sep 23 18:06:37 dist-test-slave-rvjj krb5kdc[29374](info): commencing operation
krb5kdc: starting...
W20260923 18:06:37.985699 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:06:38 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:06:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangeradmin_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangeradmin_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangerlookup_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangerlookup_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
I20260923 18:06:38.200038 18611 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/postgres -l logfile start
2026-09-23 18:06:41.518 UTC [29404] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 18:06:41.518 UTC [29404] LOG: listening on IPv4 address "127.18.44.212", port 53055
2026-09-23 18:06:41.524 UTC [29404] LOG: listening on Unix socket "/tmp/.s.PGSQL.53055"
2026-09-23 18:06:41.559 UTC [29407] LOG: database system was shut down at 2026-09-23 18:06:39 UTC
2026-09-23 18:06:41.564 UTC [29404] LOG: database system is ready to accept connections
I20260923 18:06:42.436911 18611 mini_postgres.cc:96] Postgres bound to 53055
2026-09-23 18:06:42.444 UTC [29415] FATAL: database "slave" does not exist
127.18.44.212:53055 - accepting connections
I20260923 18:06:42.445068 18611 mini_ranger.cc:161] Starting Ranger...
I20260923 18:06:42.462132 18611 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 18:06:43.163901 18611 mini_ranger.cc:87] Created ranger Postgres database
I20260923 18:06:43.164017 18611 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin
2026-09-23 18:06:43,549 [I] DB FLAVOR :POSTGRES
2026-09-23 18:06:43,549 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:06:43,549 [I] Checking connection..
2026-09-23 18:06:43,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 18:06:43,841 [I] Checking connection passed.
2026-09-23 18:06:43,842 [I] --------- Verifying version history table ---------
2026-09-23 18:06:43,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:06:44,102 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:06:44,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:06:44,373 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:06:44,374 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 18:06:44,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 18:06:44.647 UTC [29479] WARNING: there is no transaction in progress
2026-09-23 18:06:44,654 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 18:06:44,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:06:44,907 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 18:06:44,907 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:06:44,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 18:06:45,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 18:06:45,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rvjj.c.gcp-upstream.internal','N') ;"
2026-09-23 18:06:45,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:06:45,929 [I] Table x_portal_user does not exist in database ranger
2026-09-23 18:06:45,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:06:46,210 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 18:06:46,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:06:46,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:06:46,854 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 18:06:46,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 18:06:48.025 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.047 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.068 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.087 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.108 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.133 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.211 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.220 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.242 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.257 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.271 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.290 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.304 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.314 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.325 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:48.336 UTC [29587] WARNING: there is no transaction in progress
2026-09-23 18:06:49,245 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 18:06:49,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 18:06:49,508 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 18:06:49,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:06:49,776 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 18:06:49,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:06:50,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 18:06:50,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:06:50,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:06:50,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rvjj.c.gcp-upstream.internal';"
2026-09-23 18:06:50,858 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 18:06:50,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 18:06:51,120 [I] DB_PATCHES have already been applied
I20260923 18:06:51.127149 18611 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ7qt2v/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 18:06:51.127228 18611 mini_ranger.cc:193] Using host: 127.18.44.212
I20260923 18:06:51.130882 18611 mini_ranger.cc:227] Ranger admin URL: http://127.18.44.212:39019
Sep 23, 2026 6:06:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:06:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:06:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 6:06:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 6:06:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 6:06:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 6:06:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 18:06:53.132799 18611 mini_ranger.cc:160] Time spent starting Ranger: real 10.688s user 0.000s sys 0.013s
Sep 23, 2026 6:07:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 6:07:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 6:07:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 6:07:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 6:07:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 18:07:17.043684 18611 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangerkms_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/rangerkms_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.44.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.44.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 18:07:17 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 18:07:17.166075 18611 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 18:07:17.178915 18611 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 18:07:18.519008 18611 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 18:07:18.519124 18611 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms
I20260923 18:07:18.519240 18611 mini_ranger_kms.cc:227] Using postgres at 127.18.44.212:53055
2026-09-23 18:07:18,740 [I] DB FLAVOR :POSTGRES
2026-09-23 18:07:18,741 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:07:18,741 [I] Checking connection
2026-09-23 18:07:18,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 18:07:19,010 [I] connection success
2026-09-23 18:07:19,011 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 18:07:19,011 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 18:07:19,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 18:07:19,282 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 18:07:19,283 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:07:19,283 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 18:07:19,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53055/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 18:07:19,594 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 18:07:20.158485 18611 mini_ranger_kms.cc:310] Created kms service
I20260923 18:07:20.371188 18611 mini_ranger_kms.cc:326] Created kudu user
I20260923 18:07:20.418823 18611 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 18:07:20 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:20.884627 18611 mini_ranger_kms.cc:384] Added ranger policy
I20260923 18:07:20.884788 18611 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 18:07:20.884810 18611 mini_ranger_kms.cc:240] Using host: 127.18.44.212
I20260923 18:07:20.889092 18611 mini_ranger_kms.cc:276] Ranger KMS PID: 29769
I20260923 18:07:20.889228 18611 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.44.212:37261
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 6:07:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 18:07:22.956760 18611 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.791s user 0.000s sys 0.201s
I20260923 18:07:22.956897 18611 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 18:07:22.957005 18611 mini_ranger_kms.cc:401] 127.18.44.212:37261/kms/v1/keys
Sep 23 18:07:23 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790186843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 18:07:23 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
Sep 23 18:07:24 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186843, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
Sep 23 18:07:26 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:07:26.677382 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:34267
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:34267
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37261/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:07:26.860049 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:07:26.860476 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:07:26.860524 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:07:26.865662 29819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:07:26.865792 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:07:26.865811 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:07:26.865828 29819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:07:26.865844 29819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:07:26.872860 29819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.44.212:37261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:34267
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:07:26.874480 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:07:26.876036 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:07:26.888870 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:26.888899 29825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:26.889132 29819 server_base.cc:1034] running on GCE node
W20260923 18:07:26.889394 29827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:26.890326 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:07:26.891993 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:07:26.893265 29819 hybrid_clock.cc:648] HybridClock initialized: now 1790186846893247 us; error 46 us; skew 500 ppm
Sep 23 18:07:26 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186846, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:07:26.910794 29819 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:07:26.912329 29819 webserver.cc:460] Webserver started at http://127.18.44.254:42937/ using document root <none> and password file <none>
I20260923 18:07:26.913056 29819 fs_manager.cc:359] Metadata directory not provided
I20260923 18:07:26.913129 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:07:26.913383 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:07:26 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186846, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:26.983728 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "aa34911e26ed4e80b6fca391edcdd969"
format_stamp: "Formatted at 2026-09-23 18:07:26 on dist-test-slave-rvjj"
server_key: "34324e36436652c26fa8e747aa9511f9"
server_key_iv: "bdd173b09170b90a1c0ce6287ad64605"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:26.984560 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "aa34911e26ed4e80b6fca391edcdd969"
format_stamp: "Formatted at 2026-09-23 18:07:26 on dist-test-slave-rvjj"
server_key: "34324e36436652c26fa8e747aa9511f9"
server_key_iv: "bdd173b09170b90a1c0ce6287ad64605"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:26.990249 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260923 18:07:27.011052 29834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:27.012948 29819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:07:27.013154 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "aa34911e26ed4e80b6fca391edcdd969"
format_stamp: "Formatted at 2026-09-23 18:07:26 on dist-test-slave-rvjj"
server_key: "34324e36436652c26fa8e747aa9511f9"
server_key_iv: "bdd173b09170b90a1c0ce6287ad64605"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:27.013330 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:07:27.031337 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:07:27.035907 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:07:27.036141 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:07:27.045004 29819 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:34267
I20260923 18:07:27.045037 29886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:34267 every 8 connection(s)
I20260923 18:07:27.046388 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:07:27.049846 29887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:07:27.051486 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 29819
I20260923 18:07:27.051630 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:07:27.051968 18611 external_mini_cluster.cc:1428] Setting key 1e18641c694c78e84582cd6d80bf3bd3
I20260923 18:07:27.057387 29887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: Bootstrap starting.
I20260923 18:07:27.060094 29887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: Neither blocks nor log segments found. Creating new log.
I20260923 18:07:27.060945 29887 log.cc:824] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:07:27 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:07:27.063575 29887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: No bootstrap required, opened a new log
I20260923 18:07:27.068230 29887 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } }
I20260923 18:07:27.068534 29887 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:07:27.068567 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:27.053658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.18.44.254:34267)
0923 18:07:27.054212 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:27.054232 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:27.054280 (+ 48us) server_negotiation.cc:407] Connection header received
0923 18:07:27.055279 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:27.055310 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:27.055795 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:27.056206 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:07:27.057147 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:27.058667 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:27.059890 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:27.060646 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:27.060808 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:27.063451 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:07:27.063456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:07:27.063471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:07:27.063499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:07:27.065683 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:27.066145 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:27.066150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:27.066155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:27.066265 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:27.066518 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:27.066521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:27.066523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:27.066950 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:07:27.067170 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:27.067458 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 18:07:27.067759 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":180,"threads_started":1}
I20260923 18:07:27.068591 29887 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34911e26ed4e80b6fca391edcdd969, State: Initialized, Role: FOLLOWER
I20260923 18:07:27.069275 29887 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } }
I20260923 18:07:27.069437 29887 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:07:27.069522 29887 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:07:27.069693 29887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:07:27.070902 29887 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } }
I20260923 18:07:27.071332 29887 leader_election.cc:302] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa34911e26ed4e80b6fca391edcdd969; no voters:
I20260923 18:07:27.071708 29887 leader_election.cc:288] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:07:27.071866 29892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:07:27.072075 29892 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [term 1 LEADER]: Becoming Leader. State: Replica: aa34911e26ed4e80b6fca391edcdd969, State: Running, Role: LEADER
I20260923 18:07:27.072407 29892 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } }
I20260923 18:07:27.073081 29887 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:07:27.073482 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa34911e26ed4e80b6fca391edcdd969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } } }
I20260923 18:07:27.073582 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [sys.catalog]: This master's current role is: LEADER
I20260923 18:07:27.073755 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa34911e26ed4e80b6fca391edcdd969. Latest consensus state: current_term: 1 leader_uuid: "aa34911e26ed4e80b6fca391edcdd969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa34911e26ed4e80b6fca391edcdd969" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34267 } } }
I20260923 18:07:27.073817 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969 [sys.catalog]: This master's current role is: LEADER
I20260923 18:07:27.075517 29887 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:07:27.080134 29896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:07:27.085177 29896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:07:27.096997 29896 catalog_manager.cc:1347] Generated new cluster ID: 653e1104411a4fc6937ba0878bcf5bd4
I20260923 18:07:27.097079 29896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:07:27.127039 29896 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:07:27.128453 29896 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:07:27.136584 29896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: Generated new TSK 0
I20260923 18:07:27.137543 29896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:07:27.868767 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:07:28.029877 29932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:07:28.030198 29932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:07:28.030260 29932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:07:28.034752 29932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:07:28.034848 29932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:07:28.034932 29932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:07:28.041671 29932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.44.212:37261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:07:28.043740 29932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:07:28.045379 29932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:07:28.058821 29940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:28.059408 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:28.060725 29939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:28.064957 29932 server_base.cc:1034] running on GCE node
I20260923 18:07:28.065635 29932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:07:28.066551 29932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:07:28.069896 29932 hybrid_clock.cc:648] HybridClock initialized: now 1790186848069861 us; error 49 us; skew 500 ppm
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:07:28.073840 29932 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:07:28.075304 29932 webserver.cc:460] Webserver started at http://127.18.44.193:39391/ using document root <none> and password file <none>
I20260923 18:07:28.076139 29932 fs_manager.cc:359] Metadata directory not provided
I20260923 18:07:28.076296 29932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:07:28.076617 29932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:28.095743 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "c97ac537df8041deb0b37d5d3bd84d3e"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "5823bd7cfb91e8d2647fa7f9cbc47a07"
server_key_iv: "4aea824daa21acdc28b6e0aebd2b1da8"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.096956 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "c97ac537df8041deb0b37d5d3bd84d3e"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "5823bd7cfb91e8d2647fa7f9cbc47a07"
server_key_iv: "4aea824daa21acdc28b6e0aebd2b1da8"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.105144 29932 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260923 18:07:28.125329 29951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:28.127130 29932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:07:28.127403 29932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "c97ac537df8041deb0b37d5d3bd84d3e"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "5823bd7cfb91e8d2647fa7f9cbc47a07"
server_key_iv: "4aea824daa21acdc28b6e0aebd2b1da8"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.127586 29932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:07:28.187562 29932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:07:28.191426 29932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:07:28.191756 29932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:07:28.192570 29932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:07:28.194042 29932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:07:28.194170 29932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 18:07:28.194324 29932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:07:28.194420 29932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:28.206627 29932 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:34299
I20260923 18:07:28.206815 30064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:34299 every 8 connection(s)
I20260923 18:07:28.207823 29932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:07:28.216311 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 29932
I20260923 18:07:28.216444 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:07:28.216756 18611 external_mini_cluster.cc:1428] Setting key 72099756d1bbc2f84e558dd3e1ee502d
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:07:28.228610 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:28.210499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34267 (local address 127.18.44.193:60653)
0923 18:07:28.211074 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:28.211150 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:28.212447 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:28.213140 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:28.213150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:28.213559 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:07:28.214384 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:28.214407 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.215714 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.215718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:28.217263 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:28.217278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.217872 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.217878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:28.218229 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:28.219197 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:07:28.219220 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:07:28.221360 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:07:28.223866 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:28.223874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:28.223897 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:28.224549 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:28.224945 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:28.224949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:28.224951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:28.225199 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:28.225758 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:07:28.225766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:07:28.226308 (+ 542us) client_negotiation.cc:769] Sending connection context
0923 18:07:28.226692 (+ 384us) client_negotiation.cc:240] Negotiation successful
0923 18:07:28.227195 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260923 18:07:28.229818 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:28.210064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:60653 (local address 127.18.44.254:34267)
0923 18:07:28.210372 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:28.210377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:28.211826 (+ 1449us) server_negotiation.cc:407] Connection header received
0923 18:07:28.212754 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:28.212758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:28.212816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:28.212906 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:07:28.214787 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.215560 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.217400 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.217689 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.217797 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:28.221674 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:07:28.221678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:07:28.221681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:07:28.221719 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:07:28.223658 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:28.224707 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:28.224712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:28.224715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:28.224828 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:28.225321 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:28.225324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:28.225326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:28.225590 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:07:28.225692 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:28.229456 (+ 3764us) server_negotiation.cc:299] Negotiation successful
0923 18:07:28.229633 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:07:28.231156 30065 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34267
I20260923 18:07:28.231518 30065 heartbeater.cc:464] Registering TS with master...
I20260923 18:07:28.232174 30065 heartbeater.cc:511] Master 127.18.44.254:34267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:07:28.234637 29850 ts_manager.cc:194] Registered new tserver with Master: c97ac537df8041deb0b37d5d3bd84d3e (127.18.44.193:34299)
I20260923 18:07:28.236358 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:60653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:07:28.265719 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:07:28.445978 30072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:07:28.446298 30072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:07:28.446362 30072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:07:28.451828 30072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:07:28.451925 30072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:07:28.452010 30072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:07:28.458683 30072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.44.212:37261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:07:28.460363 30072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:07:28.461827 30072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:07:28.477293 30078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:28.477629 30072 server_base.cc:1034] running on GCE node
W20260923 18:07:28.477830 30077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:28.477432 30080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:28.478454 30072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:07:28.479388 30072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:07:28.480581 30072 hybrid_clock.cc:648] HybridClock initialized: now 1790186848480537 us; error 63 us; skew 500 ppm
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:07:28.484494 30072 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:07:28.485769 30072 webserver.cc:460] Webserver started at http://127.18.44.194:37613/ using document root <none> and password file <none>
I20260923 18:07:28.486519 30072 fs_manager.cc:359] Metadata directory not provided
I20260923 18:07:28.486763 30072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:07:28.487057 30072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:28.504510 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "52187f963aff4e92bd3d9cacbccb03ce"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "feed9f71081efcbf8a27f5552f0da08c"
server_key_iv: "382d81c50391c69ff473b6717d33944b"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.505573 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "52187f963aff4e92bd3d9cacbccb03ce"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "feed9f71081efcbf8a27f5552f0da08c"
server_key_iv: "382d81c50391c69ff473b6717d33944b"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.520912 30072 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.001s sys 0.004s
I20260923 18:07:28.549414 30087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:28.551323 30072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:07:28.551489 30072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "52187f963aff4e92bd3d9cacbccb03ce"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "feed9f71081efcbf8a27f5552f0da08c"
server_key_iv: "382d81c50391c69ff473b6717d33944b"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.551644 30072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:07:28.574747 30072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:07:28.578969 30072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:07:28.579288 30072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:07:28.580138 30072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:07:28.581264 30072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:07:28.581367 30072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:28.581467 30072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:07:28.581539 30072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:28.598869 30072 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:46017
I20260923 18:07:28.600373 30072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:07:28.607878 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 30072
I20260923 18:07:28.608019 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:07:28.608333 18611 external_mini_cluster.cc:1428] Setting key d4c7b55b2234d695a00ddf7f05278aa6
I20260923 18:07:28.601962 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:46017 every 8 connection(s)
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:07:28.620767 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:28.605579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:50245 (local address 127.18.44.254:34267)
0923 18:07:28.605753 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:28.605758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:28.606498 (+ 740us) server_negotiation.cc:407] Connection header received
0923 18:07:28.607843 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:28.607847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:28.607905 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:28.607958 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:07:28.609624 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.610334 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.612272 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.612610 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.612687 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:28.615935 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:07:28.615940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:07:28.615942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:07:28.615974 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:07:28.617944 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:28.618678 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:28.618682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:28.618684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:28.618798 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:28.619314 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:28.619318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:28.619320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:28.619577 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:07:28.619659 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:28.620424 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 18:07:28.620595 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:07:28.622547 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:28.604342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34267 (local address 127.18.44.194:50245)
0923 18:07:28.606307 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:28.606364 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:28.607583 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:28.608387 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:28.608397 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:28.608751 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:07:28.609449 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:28.609469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.610517 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.610520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:28.612122 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:28.612136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:28.612907 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:28.612910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:28.613117 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:28.613833 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:07:28.613854 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:07:28.615662 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:07:28.618119 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:28.618127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:28.618150 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:28.618549 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:28.618923 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:28.618926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:28.618928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:28.619172 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:28.619732 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:07:28.619740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:07:28.620128 (+ 388us) client_negotiation.cc:769] Sending connection context
0923 18:07:28.620974 (+ 846us) client_negotiation.cc:240] Negotiation successful
0923 18:07:28.621380 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1748,"thread_start_us":181,"threads_started":1}
I20260923 18:07:28.624287 30201 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34267
I20260923 18:07:28.624576 30201 heartbeater.cc:464] Registering TS with master...
I20260923 18:07:28.625301 30201 heartbeater.cc:511] Master 127.18.44.254:34267 requested a full tablet report, sending...
I20260923 18:07:28.627310 29850 ts_manager.cc:194] Registered new tserver with Master: 52187f963aff4e92bd3d9cacbccb03ce (127.18.44.194:46017)
I20260923 18:07:28.628209 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:50245
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:07:28.702958 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:07:28.872452 30206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:07:28.872952 30206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:07:28.873071 30206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:07:28.879458 30206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:07:28.879696 30206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:07:28.879837 30206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:07:28.888259 30206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.44.212:37261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:07:28.889945 30206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:07:28.891456 30206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:07:28.905406 30212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:28.906054 30211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:07:28.906313 30214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:07:28.908695 30206 server_base.cc:1034] running on GCE node
I20260923 18:07:28.909392 30206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:07:28.910220 30206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:07:28.911610 30206 hybrid_clock.cc:648] HybridClock initialized: now 1790186848911534 us; error 93 us; skew 500 ppm
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:07:28.915712 30206 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:07:28.916891 30206 webserver.cc:460] Webserver started at http://127.18.44.195:41197/ using document root <none> and password file <none>
I20260923 18:07:28.917605 30206 fs_manager.cc:359] Metadata directory not provided
I20260923 18:07:28.917695 30206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:07:28.917976 30206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:07:28 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:28.941236 30206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "7fce08a40bf14f2b900d271dbc131443"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "0ae0adc7020b42a0405f933862963597"
server_key_iv: "b4aa9613584959a1b66ced34da67a499"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.942190 30206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "7fce08a40bf14f2b900d271dbc131443"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "0ae0adc7020b42a0405f933862963597"
server_key_iv: "b4aa9613584959a1b66ced34da67a499"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:28.960824 30206 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260923 18:07:28.994120 30221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:29.001482 30206 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260923 18:07:29.001677 30206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "7fce08a40bf14f2b900d271dbc131443"
format_stamp: "Formatted at 2026-09-23 18:07:28 on dist-test-slave-rvjj"
server_key: "0ae0adc7020b42a0405f933862963597"
server_key_iv: "b4aa9613584959a1b66ced34da67a499"
server_key_version: "kuduclusterkey@0"
I20260923 18:07:29.001820 30206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:07:29.017403 30206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:07:29.021344 30206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:07:29.021620 30206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:07:29.022469 30206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:07:29.023650 30206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:07:29.023777 30206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:29.023880 30206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:07:29.023943 30206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:07:29.035954 30206 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:33931
I20260923 18:07:29.045011 30206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:07:29.045985 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 30206
I20260923 18:07:29.046120 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:07:29.046420 18611 external_mini_cluster.cc:1428] Setting key 20ca87ed2821688a6a75b91248bc1fbd
Sep 23 18:07:29 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:07:29.084517 30335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:33931 every 8 connection(s)
Sep 23 18:07:29 dist-test-slave-rvjj krb5kdc[29374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186848, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:07:29.117945 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.095089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50813 (local address 127.18.44.254:34267)
0923 18:07:29.095225 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.095229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.096720 (+ 1491us) server_negotiation.cc:407] Connection header received
0923 18:07:29.097847 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.097851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.097916 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.097992 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:07:29.099680 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.100562 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.106361 (+ 5799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.106615 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.106733 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.111947 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:07:29.111952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:07:29.111955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:07:29.111995 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:07:29.114295 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:29.115356 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:29.115361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:29.115363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:29.115610 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:29.116156 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:29.116160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:29.116162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:29.116415 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:07:29.116514 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.117576 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.117756 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:07:29.121183 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.095750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34267 (local address 127.18.44.195:50813)
0923 18:07:29.096444 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.096516 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.097590 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.098322 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.098330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.098682 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:07:29.099486 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.099527 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.103727 (+ 4200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.103733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.106147 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.106171 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.107021 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.107026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.107281 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.108460 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:07:29.108485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:07:29.110656 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:07:29.114458 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:29.114467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:29.114490 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:29.115139 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:29.115768 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:07:29.115772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:07:29.115774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:07:29.116018 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:07:29.116578 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:07:29.116586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:07:29.117252 (+ 666us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.118454 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.119372 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":214,"threads_started":1}
I20260923 18:07:29.123242 30336 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34267
I20260923 18:07:29.123587 30336 heartbeater.cc:464] Registering TS with master...
I20260923 18:07:29.124318 30336 heartbeater.cc:511] Master 127.18.44.254:34267 requested a full tablet report, sending...
I20260923 18:07:29.125730 29850 ts_manager.cc:194] Registered new tserver with Master: 7fce08a40bf14f2b900d271dbc131443 (127.18.44.195:33931)
I20260923 18:07:29.126561 29850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:50813
I20260923 18:07:29.128275 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:07:29.141179 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.130985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.18.44.254:34267)
0923 18:07:29.131180 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.131185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.132185 (+ 1000us) server_negotiation.cc:407] Connection header received
0923 18:07:29.132320 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.132324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.132379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.132478 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:07:29.133530 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.134249 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.135224 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.135521 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.135621 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.137367 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:07:29.137371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:07:29.137374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:07:29.137409 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:07:29.139226 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:29.139699 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:29.139703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:29.139704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:29.139803 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:07:29.140170 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:07:29.140174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:07:29.140175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:07:29.140439 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:07:29.140583 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.140889 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.141013 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:07:29.145011 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:07:29.148674 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:07:29.170125 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.162379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46017 (local address 127.0.0.1:53856)
0923 18:07:29.164214 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.164228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.164314 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.164945 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.164947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.164963 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:07:29.165071 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.165078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.166400 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.166403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.167889 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.167901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.169495 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.169499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.169648 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.169691 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.169805 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.169950 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":77,"threads_started":1}
I20260923 18:07:29.171103 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.162526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.18.44.194:46017)
0923 18:07:29.163739 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.163746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.164402 (+ 656us) server_negotiation.cc:407] Connection header received
0923 18:07:29.164479 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.164485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.164675 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.164803 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:07:29.165197 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.166278 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.168037 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.169174 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.169267 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.169430 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.170840 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.170964 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":130,"threads_started":1}
I20260923 18:07:29.173228 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.161084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:33931 (local address 127.0.0.1:35250)
0923 18:07:29.162835 (+ 1751us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.162890 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.162992 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.163988 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.163994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.164022 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:07:29.164181 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.164188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.167833 (+ 3645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.167836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.171201 (+ 3365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.171215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.172669 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.172673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.172769 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.172794 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.172862 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.172939 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":122,"threads_started":1}
I20260923 18:07:29.173995 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.161209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.18.44.195:33931)
0923 18:07:29.163526 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.163537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.163556 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:07:29.163633 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.163638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.163800 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.163960 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:07:29.165976 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.167690 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.171441 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.172334 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.172449 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.172590 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.173728 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.173844 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2216,"thread_start_us":112,"threads_started":1}
I20260923 18:07:29.174584 30127 tablet_service.cc:1467] Processing CreateTablet for tablet baecf872d878431f8bb06f760e57c4f5 (DEFAULT_TABLE table=test-table [id=7043de0c42094174a8c481a261104a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:07:29.175973 30127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baecf872d878431f8bb06f760e57c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:07:29.177345 30270 tablet_service.cc:1467] Processing CreateTablet for tablet baecf872d878431f8bb06f760e57c4f5 (DEFAULT_TABLE table=test-table [id=7043de0c42094174a8c481a261104a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:07:29.178565 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baecf872d878431f8bb06f760e57c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:07:29.185817 30355 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Bootstrap starting.
I20260923 18:07:29.188479 30355 tablet_bootstrap.cc:654] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Neither blocks nor log segments found. Creating new log.
I20260923 18:07:29.189308 30355 log.cc:824] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Log is configured to *not* fsync() on all Append() calls
I20260923 18:07:29.190207 30356 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Bootstrap starting.
I20260923 18:07:29.191956 30355 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: No bootstrap required, opened a new log
I20260923 18:07:29.192273 30355 ts_tablet_manager.cc:1397] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:07:29.193321 30356 tablet_bootstrap.cc:654] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Neither blocks nor log segments found. Creating new log.
I20260923 18:07:29.194345 30356 log.cc:824] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Log is configured to *not* fsync() on all Append() calls
I20260923 18:07:29.194434 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.161735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34299 (local address 127.0.0.1:47418)
0923 18:07:29.163158 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.163167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.163282 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.165890 (+ 2608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.165894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.166364 (+ 470us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:07:29.166475 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.166481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.180848 (+ 14367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.180853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.185145 (+ 4292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.185161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.193842 (+ 8681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.193848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.193978 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.194014 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.194121 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.194224 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":96,"threads_started":1}
I20260923 18:07:29.197185 30356 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: No bootstrap required, opened a new log
I20260923 18:07:29.197415 30356 ts_tablet_manager.cc:1397] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:07:29.199719 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.161889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.18.44.193:34299)
0923 18:07:29.162794 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.162802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.163290 (+ 488us) server_negotiation.cc:407] Connection header received
0923 18:07:29.163378 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.163383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.165595 (+ 2212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.165747 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:07:29.166606 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.169061 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.192572 (+ 23511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.193645 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.193822 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.194010 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.199401 (+ 5391us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.199553 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":76,"threads_started":1}
I20260923 18:07:29.201436 30356 raft_consensus.cc:348] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.201718 30356 raft_consensus.cc:374] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:07:29.201766 30356 raft_consensus.cc:729] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fce08a40bf14f2b900d271dbc131443, State: Initialized, Role: FOLLOWER
I20260923 18:07:29.202965 30356 consensus_queue.cc:260] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.203339 29997 tablet_service.cc:1467] Processing CreateTablet for tablet baecf872d878431f8bb06f760e57c4f5 (DEFAULT_TABLE table=test-table [id=7043de0c42094174a8c481a261104a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:07:29.204523 30356 ts_tablet_manager.cc:1428] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:07:29.204690 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baecf872d878431f8bb06f760e57c4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:07:29.205020 30336 heartbeater.cc:503] Master 127.18.44.254:34267 was elected leader, sending a full tablet report...
I20260923 18:07:29.199011 30355 raft_consensus.cc:348] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.215622 30355 raft_consensus.cc:374] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:07:29.219573 30355 raft_consensus.cc:729] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52187f963aff4e92bd3d9cacbccb03ce, State: Initialized, Role: FOLLOWER
I20260923 18:07:29.220371 30355 consensus_queue.cc:260] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.221809 30355 ts_tablet_manager.cc:1428] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260923 18:07:29.223029 30359 raft_consensus.cc:482] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:07:29.223284 30359 raft_consensus.cc:504] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.223776 30201 heartbeater.cc:503] Master 127.18.44.254:34267 was elected leader, sending a full tablet report...
I20260923 18:07:29.225023 30359 leader_election.cc:288] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c97ac537df8041deb0b37d5d3bd84d3e (127.18.44.193:34299), 52187f963aff4e92bd3d9cacbccb03ce (127.18.44.194:46017)
I20260923 18:07:29.230263 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.224797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46017 (local address 127.18.44.195:50091)
0923 18:07:29.224952 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.224974 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.225077 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.225994 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.225998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.226026 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:07:29.226162 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.226171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.227347 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.227351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.228600 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.228614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.229786 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.229790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.229918 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.229942 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.230029 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.230118 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 18:07:29.230193 30360 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: Bootstrap starting.
I20260923 18:07:29.231308 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.225144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50091 (local address 127.18.44.194:46017)
0923 18:07:29.225531 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.225539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.225563 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:07:29.225630 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.225633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.225686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.225783 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:07:29.226332 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.227150 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.228773 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.229471 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.229600 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.229658 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.231067 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.231159 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260923 18:07:29.232084 30153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baecf872d878431f8bb06f760e57c4f5" candidate_uuid: "7fce08a40bf14f2b900d271dbc131443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52187f963aff4e92bd3d9cacbccb03ce" is_pre_election: true
I20260923 18:07:29.232398 30153 raft_consensus.cc:2454] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fce08a40bf14f2b900d271dbc131443 in term 0.
I20260923 18:07:29.233007 30360 tablet_bootstrap.cc:654] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: Neither blocks nor log segments found. Creating new log.
I20260923 18:07:29.233065 30225 leader_election.cc:302] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52187f963aff4e92bd3d9cacbccb03ce, 7fce08a40bf14f2b900d271dbc131443; no voters:
I20260923 18:07:29.233417 30359 raft_consensus.cc:2781] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:07:29.233494 30359 raft_consensus.cc:482] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:07:29.233536 30359 raft_consensus.cc:3037] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:07:29.234028 30360 log.cc:824] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:07:29.234892 30359 raft_consensus.cc:504] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.235340 30359 leader_election.cc:288] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [CANDIDATE]: Term 1 election: Requested vote from peers c97ac537df8041deb0b37d5d3bd84d3e (127.18.44.193:34299), 52187f963aff4e92bd3d9cacbccb03ce (127.18.44.194:46017)
I20260923 18:07:29.236312 30153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baecf872d878431f8bb06f760e57c4f5" candidate_uuid: "7fce08a40bf14f2b900d271dbc131443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52187f963aff4e92bd3d9cacbccb03ce"
I20260923 18:07:29.236446 30153 raft_consensus.cc:3037] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:07:29.237260 30360 tablet_bootstrap.cc:492] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: No bootstrap required, opened a new log
I20260923 18:07:29.237473 30360 ts_tablet_manager.cc:1397] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 18:07:29.237746 30153 raft_consensus.cc:2454] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fce08a40bf14f2b900d271dbc131443 in term 1.
I20260923 18:07:29.238245 30225 leader_election.cc:302] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52187f963aff4e92bd3d9cacbccb03ce, 7fce08a40bf14f2b900d271dbc131443; no voters:
I20260923 18:07:29.239151 30359 raft_consensus.cc:2781] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:07:29.239470 30359 raft_consensus.cc:686] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 1 LEADER]: Becoming Leader. State: Replica: 7fce08a40bf14f2b900d271dbc131443, State: Running, Role: LEADER
I20260923 18:07:29.239787 30065 heartbeater.cc:503] Master 127.18.44.254:34267 was elected leader, sending a full tablet report...
I20260923 18:07:29.239832 30359 consensus_queue.cc:237] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.241860 30360 raft_consensus.cc:348] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.242157 30360 raft_consensus.cc:374] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:07:29.242213 30360 raft_consensus.cc:729] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97ac537df8041deb0b37d5d3bd84d3e, State: Initialized, Role: FOLLOWER
I20260923 18:07:29.242689 30360 consensus_queue.cc:260] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } }
I20260923 18:07:29.243898 30360 ts_tablet_manager.cc:1428] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260923 18:07:29.246794 29850 catalog_manager.cc:5495] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fce08a40bf14f2b900d271dbc131443 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "7fce08a40bf14f2b900d271dbc131443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fce08a40bf14f2b900d271dbc131443" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 } health_report { overall_health: UNKNOWN } } }
I20260923 18:07:29.251705 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.224979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:59475 (local address 127.18.44.193:34299)
0923 18:07:29.225383 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.225388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.235015 (+ 9627us) server_negotiation.cc:407] Connection header received
0923 18:07:29.235080 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.235082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.235134 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.235223 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:07:29.243920 (+ 8697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.244651 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.249245 (+ 4594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.249821 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.249949 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.250020 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.251461 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.251569 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260923 18:07:29.267863 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.224797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34299 (local address 127.18.44.195:59475)
0923 18:07:29.230456 (+ 5659us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.230471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.230562 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.240582 (+ 10020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:07:29.240586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:07:29.240614 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:07:29.240724 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.240730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.248024 (+ 7294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.248028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.249085 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:07:29.249100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.250101 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.250105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:07:29.250621 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.250657 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 18:07:29.267566 (+ 16909us) client_negotiation.cc:240] Negotiation successful
0923 18:07:29.267700 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5593,"mutex_wait_us":238,"thread_start_us":133,"threads_started":1}
I20260923 18:07:29.268966 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baecf872d878431f8bb06f760e57c4f5" candidate_uuid: "7fce08a40bf14f2b900d271dbc131443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" is_pre_election: true
I20260923 18:07:29.269320 30015 raft_consensus.cc:2454] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fce08a40bf14f2b900d271dbc131443 in term 0.
I20260923 18:07:29.269692 30016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baecf872d878431f8bb06f760e57c4f5" candidate_uuid: "7fce08a40bf14f2b900d271dbc131443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97ac537df8041deb0b37d5d3bd84d3e"
I20260923 18:07:29.269797 30016 raft_consensus.cc:3037] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:07:29.271140 30016 raft_consensus.cc:2454] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fce08a40bf14f2b900d271dbc131443 in term 1.
I20260923 18:07:29.284066 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.278275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.18.44.195:33931)
0923 18:07:29.278437 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:07:29.278443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:07:29.278465 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:07:29.278575 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:07:29.278579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:07:29.278653 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:07:29.278782 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:07:29.279339 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.280206 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.281516 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:07:29.281847 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:07:29.281975 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:07:29.282467 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:07:29.283074 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:07:29.283228 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 18:07:29.283781 (+ 553us) server_negotiation.cc:299] Negotiation successful
0923 18:07:29.283855 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:07:29.293103 30153 raft_consensus.cc:1261] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 1 FOLLOWER]: Refusing update from remote peer 7fce08a40bf14f2b900d271dbc131443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:07:29.293248 30016 raft_consensus.cc:1261] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 1 FOLLOWER]: Refusing update from remote peer 7fce08a40bf14f2b900d271dbc131443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:07:29.294278 30359 consensus_queue.cc:1035] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52187f963aff4e92bd3d9cacbccb03ce" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:07:29.294761 30359 consensus_queue.cc:1035] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97ac537df8041deb0b37d5d3bd84d3e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:07:29.323979 30368 mvcc.cc:205] Tried to move back new op lower bound from 7332605334692745216 to 7332605334491430912. Current Snapshot: MvccSnapshot[applied={T|T < 7332605334692745216}]
I20260923 18:07:29.329581 30370 mvcc.cc:205] Tried to move back new op lower bound from 7332605334692745216 to 7332605334491430912. Current Snapshot: MvccSnapshot[applied={T|T < 7332605334692745216 or (T in {7332605334692745216})}]
I20260923 18:07:29.335004 30369 mvcc.cc:205] Tried to move back new op lower bound from 7332605334692745216 to 7332605334491430912. Current Snapshot: MvccSnapshot[applied={T|T < 7332605334692745216}]
I20260923 18:07:29.339694 29850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:07:29.339905 29850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:07:29.343305 29850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa34911e26ed4e80b6fca391edcdd969: Sending DeleteTablet for 3 replicas of tablet baecf872d878431f8bb06f760e57c4f5
I20260923 18:07:29.344928 30127 tablet_service.cc:1514] Processing DeleteTablet for tablet baecf872d878431f8bb06f760e57c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:07:29 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:53856
I20260923 18:07:29.345543 29997 tablet_service.cc:1514] Processing DeleteTablet for tablet baecf872d878431f8bb06f760e57c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:07:29 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:47418
I20260923 18:07:29.346572 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet baecf872d878431f8bb06f760e57c4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:07:29 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:35250
I20260923 18:07:29.348697 30377 tablet_replica.cc:330] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: stopping tablet replica
I20260923 18:07:29.349118 30377 raft_consensus.cc:2229] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:07:29.349671 30377 raft_consensus.cc:2258] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:07:29.355717 30378 tablet_replica.cc:330] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e: stopping tablet replica
I20260923 18:07:29.356129 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 29932
I20260923 18:07:29.356138 30378 raft_consensus.cc:2229] T baecf872d878431f8bb06f760e57c4f5 P c97ac537df8041deb0b37d5d3bd84d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:07:29.358665 30377 ts_tablet_manager.cc:1905] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:07:29.362157 30377 ts_tablet_manager.cc:1918] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:07:29.371779 30377 log.cc:1196] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/baecf872d878431f8bb06f760e57c4f5
I20260923 18:07:29.372476 30377 ts_tablet_manager.cc:1939] T baecf872d878431f8bb06f760e57c4f5 P 52187f963aff4e92bd3d9cacbccb03ce: Deleting consensus metadata
W20260923 18:07:29.373080 29836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:34299 (error 108)
I20260923 18:07:29.376231 29838 catalog_manager.cc:4841] TS 52187f963aff4e92bd3d9cacbccb03ce (127.18.44.194:46017): tablet baecf872d878431f8bb06f760e57c4f5 (table test-table [id=7043de0c42094174a8c481a261104a14]) successfully deleted
I20260923 18:07:29.376729 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 30072
I20260923 18:07:29.386080 30379 tablet_replica.cc:330] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: stopping tablet replica
I20260923 18:07:29.386636 30379 raft_consensus.cc:2229] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:07:29.387246 30379 raft_consensus.cc:2258] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:07:29.389024 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:07:29.388621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:34299 (local address 127.0.0.1:47428)
0923 18:07:29.388840 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.388922 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260923 18:07:29.389298 30379 ts_tablet_manager.cc:1905] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:07:29.389619 29836 catalog_manager.cc:4584] TS c97ac537df8041deb0b37d5d3bd84d3e (127.18.44.193:34299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet baecf872d878431f8bb06f760e57c4f5: Network error: Client connection negotiation failed: client connection to 127.18.44.193:34299: connect: Connection refused (error 111)
I20260923 18:07:29.395026 30379 ts_tablet_manager.cc:1918] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:07:29.395198 30379 log.cc:1196] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/baecf872d878431f8bb06f760e57c4f5
I20260923 18:07:29.415373 30379 ts_tablet_manager.cc:1939] T baecf872d878431f8bb06f760e57c4f5 P 7fce08a40bf14f2b900d271dbc131443: Deleting consensus metadata
I20260923 18:07:29.415714 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 30206
W20260923 18:07:29.431615 30351 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:07:29.430910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:33931 (local address 127.0.0.1:35268)
0923 18:07:29.431116 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 18:07:29.431132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:07:29.431232 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:07:29.431469 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.195:33931: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 18:07:29.433010 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 29819
I20260923 18:07:29.445204 18611 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 18:07:29.791347 18611 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 18:07:29.791519 18611 mini_ranger.cc:66] Stopping Ranger...
I20260923 18:07:30.127942 18611 mini_ranger.cc:68] Stopped Ranger
2026-09-23 18:07:30.128 UTC [29404] LOG: received smart shutdown request
2026-09-23 18:07:30.132 UTC [29404] LOG: background worker "logical replication launcher" (PID 29413) exited with exit code 1
2026-09-23 18:07:30.133 UTC [29408] LOG: shutting down
2026-09-23 18:07:30.325 UTC [29404] LOG: database system is shut down
2026-09-23T18:07:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53441 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:07:30 dist-test-slave-rvjj krb5kdc[30389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:07:30 dist-test-slave-rvjj krb5kdc[30389](info): set up 2 sockets
Sep 23 18:07:30 dist-test-slave-rvjj krb5kdc[30389](info): commencing operation
krb5kdc: starting...
W20260923 18:07:31.440562 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:07:31 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:07:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangeradmin_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangeradmin_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangerlookup_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangerlookup_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
I20260923 18:07:31.660043 18611 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/postgres -l logfile start
2026-09-23 18:07:35.267 UTC [30419] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 18:07:35.267 UTC [30419] LOG: listening on IPv4 address "127.18.44.212", port 53265
2026-09-23 18:07:35.275 UTC [30419] LOG: listening on Unix socket "/tmp/.s.PGSQL.53265"
2026-09-23 18:07:35.300 UTC [30422] LOG: database system was shut down at 2026-09-23 18:07:32 UTC
2026-09-23 18:07:35.307 UTC [30419] LOG: database system is ready to accept connections
I20260923 18:07:36.220250 18611 mini_postgres.cc:96] Postgres bound to 53265
2026-09-23 18:07:36.228 UTC [30430] FATAL: database "slave" does not exist
127.18.44.212:53265 - accepting connections
I20260923 18:07:36.228916 18611 mini_ranger.cc:161] Starting Ranger...
I20260923 18:07:36.246819 18611 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 18:07:37.006618 18611 mini_ranger.cc:87] Created ranger Postgres database
I20260923 18:07:37.006723 18611 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin
2026-09-23 18:07:37,303 [I] DB FLAVOR :POSTGRES
2026-09-23 18:07:37,304 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:07:37,304 [I] Checking connection..
2026-09-23 18:07:37,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 18:07:37,605 [I] Checking connection passed.
2026-09-23 18:07:37,606 [I] --------- Verifying version history table ---------
2026-09-23 18:07:37,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:07:37,879 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:07:37,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:07:38,169 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:07:38,169 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 18:07:38,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 18:07:38.460 UTC [30494] WARNING: there is no transaction in progress
2026-09-23 18:07:38,467 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 18:07:38,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:07:38,748 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 18:07:38,748 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:07:38,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 18:07:39,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 18:07:39,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rvjj.c.gcp-upstream.internal','N') ;"
2026-09-23 18:07:39,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:07:39,941 [I] Table x_portal_user does not exist in database ranger
2026-09-23 18:07:39,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:07:40,253 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 18:07:40,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:07:40,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:07:40,814 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 18:07:40,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 18:07:42.133 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.159 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.186 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.209 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.231 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.257 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.328 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.339 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.360 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.374 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.387 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.408 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.423 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.433 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.445 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:42.458 UTC [30602] WARNING: there is no transaction in progress
2026-09-23 18:07:43,587 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 18:07:43,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 18:07:43,897 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 18:07:43,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:07:44,195 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 18:07:44,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:07:44,518 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 18:07:44,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:07:44,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:07:45,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rvjj.c.gcp-upstream.internal';"
2026-09-23 18:07:45,414 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 18:07:45,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 18:07:45,703 [I] DB_PATCHES have already been applied
I20260923 18:07:45.709420 18611 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ7qt2v/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 18:07:45.709504 18611 mini_ranger.cc:193] Using host: 127.18.44.212
I20260923 18:07:45.713258 18611 mini_ranger.cc:227] Ranger admin URL: http://127.18.44.212:49283
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 6:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 18:07:47.779587 18611 mini_ranger.cc:160] Time spent starting Ranger: real 11.551s user 0.002s sys 0.011s
Sep 23, 2026 6:08:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 6:08:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 6:08:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 6:08:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 6:08:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 18:08:12.235532 18611 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.44.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.44.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangerkms_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.44.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/rangerkms_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.44.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.44.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.44.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/HTTP_127.18.44.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 18:08:12 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186892, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 18:08:12.346818 18611 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 18:08:12.358990 18611 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 18:08:13.643544 18611 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 18:08:13.643765 18611 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms
I20260923 18:08:13.643796 18611 mini_ranger_kms.cc:227] Using postgres at 127.18.44.212:53265
2026-09-23 18:08:13,866 [I] DB FLAVOR :POSTGRES
2026-09-23 18:08:13,866 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:08:13,866 [I] Checking connection
2026-09-23 18:08:13,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 18:08:14,135 [I] connection success
2026-09-23 18:08:14,135 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 18:08:14,135 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 18:08:14,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 18:08:14,386 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 18:08:14,387 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:08:14,387 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 18:08:14,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.44.212:53265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 18:08:14,703 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 18:08:15.295195 18611 mini_ranger_kms.cc:310] Created kms service
I20260923 18:08:15.504666 18611 mini_ranger_kms.cc:326] Created kudu user
I20260923 18:08:15.552117 18611 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 18:08:15 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186892, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:15.993465 18611 mini_ranger_kms.cc:384] Added ranger policy
I20260923 18:08:15.993634 18611 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ7qt2v/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ7qt2v/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 18:08:15.993659 18611 mini_ranger_kms.cc:240] Using host: 127.18.44.212
I20260923 18:08:15.997704 18611 mini_ranger_kms.cc:276] Ranger KMS PID: 30784
I20260923 18:08:15.997813 18611 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.44.212:37161
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 6:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 18:08:18.112401 18611 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.766s user 0.006s sys 0.200s
I20260923 18:08:18.112596 18611 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 18:08:18.112751 18611 mini_ranger_kms.cc:401] 127.18.44.212:37161/kms/v1/keys
Sep 23 18:08:18 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790186898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 18:08:18 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
Sep 23 18:08:19 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.44.212@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
Sep 23 18:08:22 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:08:22.163923 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37657
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37657
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:22.293258 30834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:22.293576 30834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:22.293627 30834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:22.297919 30834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:22.298017 30834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:22.298036 30834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:22.298053 30834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:22.298066 30834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:22.304613 30834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.44.212:37161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37657
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:22.305737 30834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:22.306770 30834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:22.317948 30842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:22.317991 30839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:22.318071 30834 server_base.cc:1034] running on GCE node
W20260923 18:08:22.317981 30840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:22.319159 30834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:22.320519 30834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:22.321779 30834 hybrid_clock.cc:648] HybridClock initialized: now 1790186902321747 us; error 64 us; skew 500 ppm
Sep 23 18:08:22 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:22.325877 30834 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:22.327664 30834 webserver.cc:460] Webserver started at http://127.18.44.254:38955/ using document root <none> and password file <none>
I20260923 18:08:22.328510 30834 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:22.328606 30834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:22.328892 30834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:08:22 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186902, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:22.427539 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "28b1265071da41cbb6143c58c51f0ae9"
format_stamp: "Formatted at 2026-09-23 18:08:22 on dist-test-slave-rvjj"
server_key: "132509eb34912d245894e325707d8637"
server_key_iv: "88ac1f9182049acc0f1620ec9cd55a18"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:22.428349 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "28b1265071da41cbb6143c58c51f0ae9"
format_stamp: "Formatted at 2026-09-23 18:08:22 on dist-test-slave-rvjj"
server_key: "132509eb34912d245894e325707d8637"
server_key_iv: "88ac1f9182049acc0f1620ec9cd55a18"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:22.433147 30834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:08:22.456022 30849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:22.458588 30834 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:08:22.458866 30834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "28b1265071da41cbb6143c58c51f0ae9"
format_stamp: "Formatted at 2026-09-23 18:08:22 on dist-test-slave-rvjj"
server_key: "132509eb34912d245894e325707d8637"
server_key_iv: "88ac1f9182049acc0f1620ec9cd55a18"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:22.459085 30834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:22.490866 30834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:22.495438 30834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:22.495707 30834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:22.505980 30834 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37657
I20260923 18:08:22.506042 30901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37657 every 8 connection(s)
I20260923 18:08:22.507573 30834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:22.511150 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:22.515779 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 30834
I20260923 18:08:22.515925 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:08:22.516244 18611 external_mini_cluster.cc:1428] Setting key 390f23c11ebb070e72bec90f5a57ac1d
I20260923 18:08:22.518303 30902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Bootstrap starting.
I20260923 18:08:22.521311 30902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:22.522635 30902 log.cc:824] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:22.526273 30902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: No bootstrap required, opened a new log
Sep 23 18:08:22 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:22.532181 30902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:22.532636 30902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:22.532729 30902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b1265071da41cbb6143c58c51f0ae9, State: Initialized, Role: FOLLOWER
I20260923 18:08:22.533305 30902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:22.533607 30902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:22.533699 30902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:22.533812 30902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:22.534991 30902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:22.535552 30902 leader_election.cc:302] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b1265071da41cbb6143c58c51f0ae9; no voters:
I20260923 18:08:22.535986 30902 leader_election.cc:288] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:08:22.536217 30907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:22.536698 30907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 28b1265071da41cbb6143c58c51f0ae9, State: Running, Role: LEADER
I20260923 18:08:22.537104 30907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:22.537710 30902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:22.539208 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b1265071da41cbb6143c58c51f0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } } }
I20260923 18:08:22.539433 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:22.539925 30909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b1265071da41cbb6143c58c51f0ae9. Latest consensus state: current_term: 1 leader_uuid: "28b1265071da41cbb6143c58c51f0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } } }
I20260923 18:08:22.540030 30909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:22.540181 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:22.518110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.18.44.254:37657)
0923 18:08:22.518731 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:22.518756 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:22.518812 (+ 56us) server_negotiation.cc:407] Connection header received
0923 18:08:22.519858 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:22.519890 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:22.520434 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:22.521198 (+ 764us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:22.522522 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:22.524201 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:22.525974 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:22.526971 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:22.527193 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:22.531582 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:22.531591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:22.531619 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:22.531677 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:22.534828 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:22.535729 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:22.535740 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:22.535761 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:22.535965 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:22.536607 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:22.536613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:22.536615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:22.537501 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:22.537758 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:22.538119 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 18:08:22.538581 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":147,"threads_started":1}
I20260923 18:08:22.540339 30902 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:08:22.541739 30918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:22.549293 30918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:22.560612 30918 catalog_manager.cc:1347] Generated new cluster ID: 00c4bd70457141b7bc15743fda1a1194
I20260923 18:08:22.560736 30918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:22.572912 30918 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:08:22.574496 30918 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:22.592368 30918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Generated new TSK 0
I20260923 18:08:22.593389 30918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:08:23.428303 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:23.577459 30947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:23.577775 30947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:23.577836 30947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:23.582234 30947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:23.582327 30947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:23.582425 30947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:23.588438 30947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:23.589701 30947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:23.591034 30947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:23.602854 30954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:23.602810 30955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:23.603083 30947 server_base.cc:1034] running on GCE node
W20260923 18:08:23.602856 30957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:23.603969 30947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:23.604852 30947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:23.606063 30947 hybrid_clock.cc:648] HybridClock initialized: now 1790186903606029 us; error 52 us; skew 500 ppm
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:23.609967 30947 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:23.611243 30947 webserver.cc:460] Webserver started at http://127.18.44.193:44653/ using document root <none> and password file <none>
I20260923 18:08:23.612118 30947 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:23.612258 30947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:23.612528 30947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:23.629287 30947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "4c288621d2a3475c8e8602e25c85a538"
format_stamp: "Formatted at 2026-09-23 18:08:23 on dist-test-slave-rvjj"
server_key: "e5a4bca920230c9d01421ae6bb76f8da"
server_key_iv: "5ee0cbb2c7adab628b5c6674329b8c06"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:23.630223 30947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "4c288621d2a3475c8e8602e25c85a538"
format_stamp: "Formatted at 2026-09-23 18:08:23 on dist-test-slave-rvjj"
server_key: "e5a4bca920230c9d01421ae6bb76f8da"
server_key_iv: "5ee0cbb2c7adab628b5c6674329b8c06"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:23.635486 30947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 18:08:23.663825 30964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:23.665974 30947 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260923 18:08:23.666157 30947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "4c288621d2a3475c8e8602e25c85a538"
format_stamp: "Formatted at 2026-09-23 18:08:23 on dist-test-slave-rvjj"
server_key: "e5a4bca920230c9d01421ae6bb76f8da"
server_key_iv: "5ee0cbb2c7adab628b5c6674329b8c06"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:23.666327 30947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:23.708415 30947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:23.712497 30947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:23.712729 30947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:23.713480 30947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:23.714702 30947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:23.714774 30947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:23.714833 30947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:23.714910 30947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 18:08:23.727535 30947 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:33901
I20260923 18:08:23.727587 31079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:33901 every 8 connection(s)
I20260923 18:08:23.728961 30947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:23.738217 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 30947
I20260923 18:08:23.738348 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:08:23.738642 18611 external_mini_cluster.cc:1428] Setting key cf8e96830a0926b72b6830cc915cd2f0
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:23.751713 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:23.731978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.193:58851)
0923 18:08:23.732617 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:23.732681 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:23.733785 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:23.736776 (+ 2991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:23.736789 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:23.737223 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:23.738102 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:23.738133 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:23.739450 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:23.739455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:23.741039 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:23.741055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:23.741802 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:23.741807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:23.742056 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:23.742822 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:23.742848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:23.744733 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:23.747224 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:23.747232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:23.747286 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:23.747743 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:23.748143 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:23.748147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:23.748149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:23.748483 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:23.749070 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:23.749079 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:23.749505 (+ 426us) client_negotiation.cc:769] Sending connection context
0923 18:08:23.749734 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 18:08:23.750243 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":181,"threads_started":1}
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:08:23.753104 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:23.731403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:58851 (local address 127.18.44.254:37657)
0923 18:08:23.736230 (+ 4827us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:23.736235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:23.736253 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:08:23.736306 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:23.736310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:23.736369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:23.736490 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:23.738364 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:23.739208 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:23.741225 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:23.741596 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:23.742467 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:23.745016 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:23.745021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:23.745024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:23.745055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:23.747034 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:23.747911 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:23.747915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:23.747917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:23.748023 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:23.748633 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:23.748637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:23.748638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:23.748910 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:23.752571 (+ 3661us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:23.752708 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 18:08:23.752895 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4723,"thread_start_us":91,"threads_started":1}
I20260923 18:08:23.754546 31080 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:23.754863 31080 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:23.755545 31080 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:08:23.757508 30866 ts_manager.cc:194] Registered new tserver with Master: 4c288621d2a3475c8e8602e25c85a538 (127.18.44.193:33901)
I20260923 18:08:23.759142 30866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:58851
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:08:23.779299 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:23.946055 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:23.946482 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:23.946550 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:23.953261 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:23.953362 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:23.953452 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:23.962044 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:23.963590 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:23.965019 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:23.980928 31095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:23.981343 31093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:23.981469 31092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:23.982762 31086 server_base.cc:1034] running on GCE node
I20260923 18:08:23.983471 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:23.984359 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:23.987675 31086 hybrid_clock.cc:648] HybridClock initialized: now 1790186903987469 us; error 211 us; skew 500 ppm
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:23.992332 31086 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:23.994283 31086 webserver.cc:460] Webserver started at http://127.18.44.194:45689/ using document root <none> and password file <none>
I20260923 18:08:23.995440 31086 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:23.995571 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:23.995862 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:08:23 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:24.017467 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "ff8c073e89804662a3d91ff158af4adf"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "5530f5a31a2286de7a43cb0b56714097"
server_key_iv: "4d550fafafdb449135516b2b4122f0bb"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.018309 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "ff8c073e89804662a3d91ff158af4adf"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "5530f5a31a2286de7a43cb0b56714097"
server_key_iv: "4d550fafafdb449135516b2b4122f0bb"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.023717 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:08:24.043728 31102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.045608 31086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:24.045773 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "ff8c073e89804662a3d91ff158af4adf"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "5530f5a31a2286de7a43cb0b56714097"
server_key_iv: "4d550fafafdb449135516b2b4122f0bb"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.045943 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:24.100590 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:24.104569 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:24.104790 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:24.105593 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:24.106688 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:24.106756 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.106817 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:24.106845 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.117973 31086 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:35147
I20260923 18:08:24.119210 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:24.124151 31215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:35147 every 8 connection(s)
I20260923 18:08:24.127162 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31086
I20260923 18:08:24.127358 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:08:24.127820 18611 external_mini_cluster.cc:1428] Setting key 7f1adf893008acf45069e1217c5b6abd
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:24.141444 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:24.123535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:39797 (local address 127.18.44.254:37657)
0923 18:08:24.123677 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:24.123682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:24.125249 (+ 1567us) server_negotiation.cc:407] Connection header received
0923 18:08:24.126681 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:24.126685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:24.126867 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:24.127031 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:24.129248 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.130066 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.132419 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.132784 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.132899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:24.136394 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:24.136399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:24.136403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:24.136437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:24.138340 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:24.139233 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:24.139237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:24.139239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:24.139407 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:24.139940 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:24.139944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:24.139946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:24.140196 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:24.140305 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:24.141092 (+ 787us) server_negotiation.cc:299] Negotiation successful
0923 18:08:24.141262 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:08:24.143388 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:24.122991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.194:39797)
0923 18:08:24.125021 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:24.125093 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:24.126350 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:24.127767 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:24.127778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:24.128207 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:24.129040 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:24.129066 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.130309 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.130313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:24.132241 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:24.132258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.132996 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.133002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:24.133250 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:24.133977 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:24.134003 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:24.136115 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:24.138521 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:24.138531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:24.138554 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:24.139064 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:24.139522 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:24.139526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:24.139528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:24.139810 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:24.140340 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:24.140349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:24.140745 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 18:08:24.141731 (+ 986us) client_negotiation.cc:240] Negotiation successful
0923 18:08:24.142153 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"thread_start_us":627,"threads_started":1}
I20260923 18:08:24.145045 31216 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:24.145345 31216 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:24.146507 31216 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:24.147894 30866 ts_manager.cc:194] Registered new tserver with Master: ff8c073e89804662a3d91ff158af4adf (127.18.44.194:35147)
I20260923 18:08:24.148703 30866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:39797
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:08:24.171847 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186903, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
W20260923 18:08:24.334436 31221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:24.335218 31221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:24.335523 31221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:24.343356 31221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:24.343551 31221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:24.343714 31221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:24.352346 31221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:24.354138 31221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:24.355680 31221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:24.369517 31227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:24.370388 31226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:24.371340 31221 server_base.cc:1034] running on GCE node
W20260923 18:08:24.371610 31229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:24.372215 31221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:24.373066 31221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:24.374420 31221 hybrid_clock.cc:648] HybridClock initialized: now 1790186904374355 us; error 91 us; skew 500 ppm
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186904, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:24.378263 31221 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:24.379520 31221 webserver.cc:460] Webserver started at http://127.18.44.195:35829/ using document root <none> and password file <none>
I20260923 18:08:24.380290 31221 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:24.380431 31221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:24.380710 31221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186904, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:24.425361 31221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "2923897b74d7459d9b4851212a810f68"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "a09e8c96923b7c4f198c173830382d51"
server_key_iv: "9e91e6eb66d8281df4e9ba624f24c597"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.426347 31221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "2923897b74d7459d9b4851212a810f68"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "a09e8c96923b7c4f198c173830382d51"
server_key_iv: "9e91e6eb66d8281df4e9ba624f24c597"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.434190 31221 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260923 18:08:24.457845 31236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.459832 31221 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:08:24.459998 31221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "2923897b74d7459d9b4851212a810f68"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "a09e8c96923b7c4f198c173830382d51"
server_key_iv: "9e91e6eb66d8281df4e9ba624f24c597"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.460217 31221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:24.487027 31221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:24.491495 31221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:24.491791 31221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:24.492717 31221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:24.493950 31221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:24.494081 31221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.494199 31221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:24.494269 31221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.505088 31221 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46651
I20260923 18:08:24.506458 31221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:24.507670 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31221
I20260923 18:08:24.507781 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:08:24.508098 18611 external_mini_cluster.cc:1428] Setting key 8ab4a6bcb811566533a63d121a12077b
I20260923 18:08:24.517333 31349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46651 every 8 connection(s)
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186904, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:24.543381 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:24.519520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:46489 (local address 127.18.44.254:37657)
0923 18:08:24.519665 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:24.519670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:24.522720 (+ 3050us) server_negotiation.cc:407] Connection header received
0923 18:08:24.524272 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:24.524277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:24.524347 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:24.524430 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:24.526419 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.529396 (+ 2977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.531635 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.531989 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.532088 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:24.535718 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:24.535722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:24.535725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:24.535762 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:24.537527 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:24.538331 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:24.538335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:24.538337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:24.538447 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:24.538929 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:24.538932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:24.538934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:24.539193 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:24.539341 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:24.542948 (+ 3607us) server_negotiation.cc:299] Negotiation successful
0923 18:08:24.543158 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:08:24.545747 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:24.518273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.195:46489)
0923 18:08:24.522355 (+ 4082us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:24.522447 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:24.523931 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:24.524998 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:24.525010 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:24.525448 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:24.526201 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:24.526221 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.529637 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.529641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:24.531425 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:24.531448 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:24.532343 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:24.532348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:24.532620 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:24.533445 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:24.533470 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:24.535438 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:24.537728 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:24.537735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:24.537758 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:24.538192 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:24.538565 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:24.538568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:24.538570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:24.538795 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:24.541822 (+ 3027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:24.541834 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:24.542555 (+ 721us) client_negotiation.cc:769] Sending connection context
0923 18:08:24.544006 (+ 1451us) client_negotiation.cc:240] Negotiation successful
0923 18:08:24.544557 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3770,"thread_start_us":186,"threads_started":1}
I20260923 18:08:24.547816 31350 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:24.548125 31350 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:24.548825 31350 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:24.552841 30866 ts_manager.cc:194] Registered new tserver with Master: 2923897b74d7459d9b4851212a810f68 (127.18.44.195:46651)
I20260923 18:08:24.553897 30866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:46489
I20260923 18:08:24.561691 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:08:24.562568 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 30947
I20260923 18:08:24.576659 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31086
I20260923 18:08:24.584511 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31221
I20260923 18:08:24.591861 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 30834
I20260923 18:08:24.598560 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37657
--webserver_interface=127.18.44.254
--webserver_port=38955
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:37657
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:24.716925 31353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:24.717252 31353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:24.717309 31353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:24.721647 31353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:24.721740 31353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:24.721766 31353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:24.721786 31353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:24.721807 31353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:24.728003 31353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:37657
--ranger_config_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=38955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:24.729218 31353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:24.730274 31353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:24.739497 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:24.739563 31359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:24.739990 31353 server_base.cc:1034] running on GCE node
W20260923 18:08:24.739497 31361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:24.740737 31353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:24.741990 31353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:24.743175 31353 hybrid_clock.cc:648] HybridClock initialized: now 1790186904743153 us; error 46 us; skew 500 ppm
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186904, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:24.752218 31353 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:24.753446 31353 webserver.cc:460] Webserver started at http://127.18.44.254:38955/ using document root <none> and password file <none>
I20260923 18:08:24.754053 31353 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:24.754140 31353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:08:24 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186904, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:24.860013 31353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:08:24.862586 31369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:24.864187 31353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:08:24.864362 31353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "28b1265071da41cbb6143c58c51f0ae9"
format_stamp: "Formatted at 2026-09-23 18:08:22 on dist-test-slave-rvjj"
server_key: "132509eb34912d245894e325707d8637"
server_key_iv: "88ac1f9182049acc0f1620ec9cd55a18"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:24.865048 31353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:24.886516 31353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:24.889639 31353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:24.889853 31353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:24.898125 31353 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:37657
I20260923 18:08:24.898159 31421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:37657 every 8 connection(s)
I20260923 18:08:24.899421 31353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:24.906790 31422 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:08:24.908283 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31353
I20260923 18:08:24.908799 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:33901
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:37657
--webserver_port=44653
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:24.909554 31422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Bootstrap starting.
I20260923 18:08:24.934099 31422 log.cc:824] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:24.943645 31422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:08:24.944147 31422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Bootstrap complete.
I20260923 18:08:24.949793 31422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:24.950341 31422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b1265071da41cbb6143c58c51f0ae9, State: Initialized, Role: FOLLOWER
I20260923 18:08:24.950945 31422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:24.951099 31422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:24.951174 31422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:24.951292 31422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:08:24.954094 31422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:24.954531 31422 leader_election.cc:302] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b1265071da41cbb6143c58c51f0ae9; no voters:
I20260923 18:08:24.954936 31422 leader_election.cc:288] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:08:24.955176 31426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:08:24.955505 31426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [term 2 LEADER]: Becoming Leader. State: Replica: 28b1265071da41cbb6143c58c51f0ae9, State: Running, Role: LEADER
I20260923 18:08:24.955921 31426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } }
I20260923 18:08:24.956478 31422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:24.957165 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28b1265071da41cbb6143c58c51f0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } } }
I20260923 18:08:24.957377 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:24.957633 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b1265071da41cbb6143c58c51f0ae9. Latest consensus state: current_term: 2 leader_uuid: "28b1265071da41cbb6143c58c51f0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b1265071da41cbb6143c58c51f0ae9" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 37657 } } }
I20260923 18:08:24.957763 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:24.958068 31435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:24.961676 31435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:24.962291 31422 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:08:24.962590 31435 catalog_manager.cc:1259] Loaded cluster ID: 00c4bd70457141b7bc15743fda1a1194
I20260923 18:08:24.962695 31435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:24.965314 31435 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:24.966228 31435 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Loaded TSK: 0
I20260923 18:08:24.967756 31435 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:08:25.077777 31424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:25.078200 31424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:25.078284 31424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:25.085076 31424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:25.085187 31424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:25.085280 31424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:25.093917 31424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:33901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=44653
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:25.095481 31424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:25.096906 31424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:25.110839 31459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:25.110891 31460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:25.111135 31462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:25.111443 31424 server_base.cc:1034] running on GCE node
I20260923 18:08:25.111974 31424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:25.112860 31424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:25.114069 31424 hybrid_clock.cc:648] HybridClock initialized: now 1790186905114030 us; error 59 us; skew 500 ppm
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:25.117918 31424 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:25.121172 31424 webserver.cc:460] Webserver started at http://127.18.44.193:44653/ using document root <none> and password file <none>
I20260923 18:08:25.122090 31424 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:25.122300 31424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:25.152674 31424 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:08:25.156073 31470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.157876 31424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 18:08:25.158041 31424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "4c288621d2a3475c8e8602e25c85a538"
format_stamp: "Formatted at 2026-09-23 18:08:23 on dist-test-slave-rvjj"
server_key: "e5a4bca920230c9d01421ae6bb76f8da"
server_key_iv: "5ee0cbb2c7adab628b5c6674329b8c06"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:25.158805 31424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:25.174973 31424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:25.179068 31424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:25.179368 31424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:25.180142 31424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:25.181289 31424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:25.181354 31424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.181409 31424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:25.181442 31424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.193212 31424 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:33901
I20260923 18:08:25.194780 31424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:25.195109 31583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:33901 every 8 connection(s)
I20260923 18:08:25.195328 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31424
I20260923 18:08:25.195840 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:35147
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:37657
--webserver_port=45689
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:25.225935 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.203146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.193:34323)
0923 18:08:25.204803 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:25.204878 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:25.207727 (+ 2849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:25.210216 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:25.210224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:25.210610 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:25.211218 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.211236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.213222 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.213226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.214195 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.214208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.215681 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.215686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.215920 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.216724 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:25.216747 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:25.218662 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:25.221720 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.221728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.221746 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.222135 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.222643 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.222647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.222650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.222858 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.223869 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:25.223878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:25.224213 (+ 335us) client_negotiation.cc:769] Sending connection context
0923 18:08:25.224458 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 18:08:25.224852 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":151,"threads_started":1}
I20260923 18:08:25.227839 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.203144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:34323 (local address 127.18.44.254:37657)
0923 18:08:25.203901 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:25.203926 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:25.205732 (+ 1806us) server_negotiation.cc:407] Connection header received
0923 18:08:25.208601 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:25.208618 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:25.209157 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:25.209585 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:25.211447 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.213045 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.214365 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.215182 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.215417 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.218968 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:25.218976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:25.218997 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:25.219038 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:25.221539 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.222299 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.222306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.222312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.222505 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.222980 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.222983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.222985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.223661 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:25.223805 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:25.226384 (+ 2579us) server_negotiation.cc:299] Negotiation successful
0923 18:08:25.226768 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":151,"threads_started":1}
I20260923 18:08:25.238374 31585 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:25.238694 31585 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.239389 31585 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.242223 31585 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:37657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 18:08:25.242422 31585 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.242640 31585 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:25.243981 31585 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.244223 31585 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.245241 31585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:08:25.368697 31584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:25.369129 31584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:25.369210 31584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:25.375747 31584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:25.375846 31584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:25.375937 31584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:25.383441 31584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:35147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=45689
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:25.384590 31584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:25.385967 31584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:25.400216 31593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:25.400233 31594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:25.400667 31596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:25.400667 31584 server_base.cc:1034] running on GCE node
I20260923 18:08:25.401285 31584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:25.402041 31584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:25.403293 31584 hybrid_clock.cc:648] HybridClock initialized: now 1790186905403228 us; error 41 us; skew 500 ppm
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:25.406697 31584 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:25.407750 31584 webserver.cc:460] Webserver started at http://127.18.44.194:45689/ using document root <none> and password file <none>
I20260923 18:08:25.408468 31584 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:25.408583 31584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:25.429563 31584 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:25.432834 31604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.434352 31584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:25.434525 31584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "ff8c073e89804662a3d91ff158af4adf"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "5530f5a31a2286de7a43cb0b56714097"
server_key_iv: "4d550fafafdb449135516b2b4122f0bb"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:25.435424 31584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:25.455340 31584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:25.459612 31584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:25.459951 31584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:25.460920 31584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:25.462208 31584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:25.462333 31584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.462481 31584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:25.462553 31584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.475795 31584 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:35147
I20260923 18:08:25.477437 31584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:25.478513 31717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:35147 every 8 connection(s)
I20260923 18:08:25.486114 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31584
I20260923 18:08:25.486593 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.44.212:37161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:46651
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:37657
--webserver_port=35829
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:25.508488 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.485462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:55869 (local address 127.18.44.254:37657)
0923 18:08:25.485711 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:25.485716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:25.487977 (+ 2261us) server_negotiation.cc:407] Connection header received
0923 18:08:25.489247 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:25.489250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:25.489316 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:25.489396 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:25.491425 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.492284 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.494417 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.494749 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.494865 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.502583 (+ 7718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:25.502589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:25.502593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:25.502643 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:25.505234 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.506303 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.506307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.506309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.506438 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.506939 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.506942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.506944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.507200 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:25.507325 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:25.508128 (+ 803us) server_negotiation.cc:299] Negotiation successful
0923 18:08:25.508295 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260923 18:08:25.510247 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.486008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.194:55869)
0923 18:08:25.487741 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:25.487822 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:25.488992 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:25.489758 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:25.489769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:25.490345 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:25.491158 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.491184 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.492495 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.492499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.494209 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.494228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.495130 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.495133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.495390 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.496238 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:25.496262 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:25.498614 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:25.505600 (+ 6986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.505612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.505637 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.506125 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.506582 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.506585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.506587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.506821 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.507409 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:25.507417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:25.507788 (+ 371us) client_negotiation.cc:769] Sending connection context
0923 18:08:25.508757 (+ 969us) client_negotiation.cc:240] Negotiation successful
0923 18:08:25.509168 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":199,"threads_started":1}
I20260923 18:08:25.511688 31718 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:25.511965 31718 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.512591 31718 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.514564 31718 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:37657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 18:08:25.514714 31718 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.515043 31718 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:25.516235 31718 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.516455 31718 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.517308 31718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:08:25.650354 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:25.650797 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:25.650866 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:25.656791 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:25.656919 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:25.657018 31721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:25.665388 31721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.44.212:37161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:46651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=35829
--tserver_master_addrs=127.18.44.254:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:25.666709 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:25.668067 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:25.680094 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:25.680141 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:25.681207 31721 server_base.cc:1034] running on GCE node
W20260923 18:08:25.684046 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:25.684784 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:25.685809 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:25.687042 31721 hybrid_clock.cc:648] HybridClock initialized: now 1790186905687008 us; error 48 us; skew 500 ppm
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:25.691476 31721 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:25.692582 31721 webserver.cc:460] Webserver started at http://127.18.44.195:35829/ using document root <none> and password file <none>
I20260923 18:08:25.693840 31721 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:25.694124 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:25.720476 31721 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:08:25.723223 31738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.725062 31721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:25.725217 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "2923897b74d7459d9b4851212a810f68"
format_stamp: "Formatted at 2026-09-23 18:08:24 on dist-test-slave-rvjj"
server_key: "a09e8c96923b7c4f198c173830382d51"
server_key_iv: "9e91e6eb66d8281df4e9ba624f24c597"
server_key_version: "kuduclusterkey@0"
I20260923 18:08:25.725833 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:25.769678 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:25.773518 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:25.773751 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:25.774585 31721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:25.775550 31721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:25.775600 31721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.775642 31721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:25.775671 31721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:25.787464 31721 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:46651
I20260923 18:08:25.788107 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:46651 every 8 connection(s)
I20260923 18:08:25.788898 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:25.791903 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31721
Sep 23 18:08:25 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186905, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:25.811812 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.792667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.18.44.254:37657)
0923 18:08:25.795605 (+ 2938us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:25.795616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:25.795628 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:08:25.795658 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:25.795660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:25.795689 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:25.795766 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:25.796637 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.797355 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.803390 (+ 6035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.803863 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.804048 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.806497 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:25.806501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:25.806504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:25.806545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:25.808544 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.809184 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.809188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.809190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.809298 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.809641 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.809643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.809644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.809882 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:25.811203 (+ 1321us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:25.811327 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 18:08:25.811602 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2849,"thread_start_us":94,"threads_started":1}
I20260923 18:08:25.814450 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.791543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:52573 (local address 127.18.44.254:37657)
0923 18:08:25.792892 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:25.792897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:25.794331 (+ 1434us) server_negotiation.cc:407] Connection header received
0923 18:08:25.795359 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:25.795363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:25.795434 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:25.795518 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:25.797125 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.797837 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.799784 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.800155 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.800280 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.804033 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:25.804038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:25.804041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:25.804077 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:25.806964 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.811935 (+ 4971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.811939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.811940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.812046 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:25.812609 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:25.812612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:25.812614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:25.812858 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:25.812983 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:25.814069 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0923 18:08:25.814285 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248}
I20260923 18:08:25.815481 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:25.791760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:37657 (local address 127.18.44.195:52573)
0923 18:08:25.794150 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:25.794207 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:25.795098 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:25.795776 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:25.795788 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:25.796192 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:25.796928 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.796951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.797984 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.797987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.799605 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:25.799622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:25.800373 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:25.800378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:25.800635 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:25.801459 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:25.801484 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:25.803749 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:25.811232 (+ 7483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.811242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.811307 (+ 65us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.811787 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.812209 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:25.812213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:25.812215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:25.812452 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:25.812992 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:25.813001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:25.813683 (+ 682us) client_negotiation.cc:769] Sending connection context
0923 18:08:25.814085 (+ 402us) client_negotiation.cc:240] Negotiation successful
0923 18:08:25.814485 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"thread_start_us":176,"threads_started":1}
I20260923 18:08:25.816802 31852 heartbeater.cc:347] Connected to a master server at 127.18.44.254:37657
I20260923 18:08:25.817085 31852 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.817662 31852 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.819393 31852 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:37657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 18:08:25.819547 31852 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.819777 31852 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:25.820883 31852 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:25.821113 31852 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
W20260923 18:08:25.822100 31852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 18:08:26 dist-test-slave-rvjj krb5kdc[30389](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790186906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:26.245574 31585 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:26.245906 31585 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:26.247172 31386 ts_manager.cc:194] Registered new tserver with Master: 4c288621d2a3475c8e8602e25c85a538 (127.18.44.193:33901)
I20260923 18:08:26.248951 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:34323
I20260923 18:08:26.517593 31718 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:26.517956 31718 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:26.518999 31386 ts_manager.cc:194] Registered new tserver with Master: ff8c073e89804662a3d91ff158af4adf (127.18.44.194:35147)
I20260923 18:08:26.519979 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:55869
Sep 23 18:08:26 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:26.822356 31852 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:26.822700 31852 heartbeater.cc:511] Master 127.18.44.254:37657 requested a full tablet report, sending...
I20260923 18:08:26.824498 31386 ts_manager.cc:194] Registered new tserver with Master: 2923897b74d7459d9b4851212a810f68 (127.18.44.195:46651)
I20260923 18:08:26.825599 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:52573
I20260923 18:08:26.836264 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:08:26.848683 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.838445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.18.44.254:37657)
0923 18:08:26.838754 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:26.838759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:26.838919 (+ 160us) server_negotiation.cc:407] Connection header received
0923 18:08:26.839067 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:26.839071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:26.839203 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:26.839357 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:26.840235 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.841250 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.842484 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.842857 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.842982 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.844768 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:26.844773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:26.844777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:26.844814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:26.846680 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:26.847239 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:26.847243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:26.847245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:26.847393 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:26.847672 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:26.847674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:26.847676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:26.847958 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:26.848104 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:26.848330 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 18:08:26.848481 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260923 18:08:26.853538 31386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:08:26.859066 31386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:08:26.882740 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.872606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35147 (local address 127.0.0.1:59512)
0923 18:08:26.875110 (+ 2504us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:26.875167 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:26.875324 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:26.877396 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:26.877486 (+ 90us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:26.877511 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:26.877793 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.877799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.879321 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.879325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.880771 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.880783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.882152 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.882156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.882277 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.882316 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 18:08:26.882436 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 18:08:26.882572 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2324,"mutex_wait_us":160,"thread_start_us":114,"threads_started":1}
I20260923 18:08:26.883453 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.872713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.18.44.194:35147)
0923 18:08:26.876744 (+ 4031us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:26.876751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:26.876774 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:08:26.876899 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:26.876905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:26.877096 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:26.877228 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:26.877924 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.879151 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.880929 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.881801 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.881913 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.882082 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:26.883155 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0923 18:08:26.883306 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3919,"thread_start_us":53,"threads_started":1}
I20260923 18:08:26.885037 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.874634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33901 (local address 127.0.0.1:60350)
0923 18:08:26.875587 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:26.875599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:26.876481 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:26.879652 (+ 3171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:26.879655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:26.879678 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:26.879772 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.879778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.881400 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.881403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.883507 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.883519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.884654 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.884658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.884752 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.884777 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:08:26.884822 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 18:08:26.884889 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":106,"threads_started":1}
I20260923 18:08:26.885690 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.875947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.18.44.193:33901)
0923 18:08:26.877824 (+ 1877us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:26.877830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:26.877855 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:08:26.877949 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:26.877953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:26.878432 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:26.879935 (+ 1503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:26.880077 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.881260 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.883666 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.884510 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.885247 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.885406 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:26.885459 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 18:08:26.885554 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":103,"threads_started":1}
I20260923 18:08:26.888029 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.875006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46651 (local address 127.0.0.1:58694)
0923 18:08:26.876311 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:26.876326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:26.876434 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:26.877402 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:26.877409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:26.877436 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:26.877654 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.877663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.885110 (+ 7447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.885114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.886188 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:26.886200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.887603 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.887607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:26.887697 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.887721 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:08:26.887800 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 18:08:26.887884 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260923 18:08:26.888468 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:26.875516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.18.44.195:46651)
0923 18:08:26.876682 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:26.876689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:26.876713 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:08:26.876794 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:26.876799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:26.877033 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:26.877185 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:26.883842 (+ 6657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.884997 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.886346 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:26.887279 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:26.887389 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:26.887535 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:26.888228 (+ 693us) server_negotiation.cc:299] Negotiation successful
0923 18:08:26.888336 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":113,"threads_started":1}
I20260923 18:08:26.888751 31634 tablet_service.cc:1467] Processing CreateTablet for tablet 99049260e2274de2a9bc27e4fa9e4e15 (DEFAULT_TABLE table=test-table [id=25b4d429940940a7b1fe0290476d76ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:26.889710 31634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99049260e2274de2a9bc27e4fa9e4e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:26.889351 31518 tablet_service.cc:1467] Processing CreateTablet for tablet 99049260e2274de2a9bc27e4fa9e4e15 (DEFAULT_TABLE table=test-table [id=25b4d429940940a7b1fe0290476d76ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:26.890332 31518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99049260e2274de2a9bc27e4fa9e4e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:26.894297 31767 tablet_service.cc:1467] Processing CreateTablet for tablet 99049260e2274de2a9bc27e4fa9e4e15 (DEFAULT_TABLE table=test-table [id=25b4d429940940a7b1fe0290476d76ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:26.895619 31767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99049260e2274de2a9bc27e4fa9e4e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:26.909065 31887 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Bootstrap starting.
I20260923 18:08:26.911825 31887 tablet_bootstrap.cc:654] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:26.912868 31887 log.cc:824] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:26.917227 31887 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: No bootstrap required, opened a new log
I20260923 18:08:26.917471 31887 ts_tablet_manager.cc:1397] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260923 18:08:26.919826 31886 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Bootstrap starting.
I20260923 18:08:26.923089 31886 tablet_bootstrap.cc:654] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:26.924494 31886 log.cc:824] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:26.924316 31887 raft_consensus.cc:348] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:26.924711 31887 raft_consensus.cc:374] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:26.924793 31887 raft_consensus.cc:729] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c288621d2a3475c8e8602e25c85a538, State: Initialized, Role: FOLLOWER
I20260923 18:08:26.925384 31887 consensus_queue.cc:260] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:26.927410 31887 ts_tablet_manager.cc:1428] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260923 18:08:26.927914 31886 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: No bootstrap required, opened a new log
I20260923 18:08:26.927958 31585 heartbeater.cc:503] Master 127.18.44.254:37657 was elected leader, sending a full tablet report...
I20260923 18:08:26.928162 31886 ts_tablet_manager.cc:1397] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 18:08:26.924052 31889 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Bootstrap starting.
I20260923 18:08:26.932729 31889 tablet_bootstrap.cc:654] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:26.932890 31886 raft_consensus.cc:348] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:26.933238 31886 raft_consensus.cc:374] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:26.933319 31886 raft_consensus.cc:729] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8c073e89804662a3d91ff158af4adf, State: Initialized, Role: FOLLOWER
I20260923 18:08:26.933727 31889 log.cc:824] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:26.933910 31886 consensus_queue.cc:260] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
Sep 23 18:08:26 dist-test-slave-rvjj krb5kdc[30389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186906, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for HTTP/127.18.44.212@KRBTEST.COM
I20260923 18:08:26.935833 31886 ts_tablet_manager.cc:1428] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260923 18:08:26.936269 31718 heartbeater.cc:503] Master 127.18.44.254:37657 was elected leader, sending a full tablet report...
I20260923 18:08:26.943835 31889 tablet_bootstrap.cc:492] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: No bootstrap required, opened a new log
I20260923 18:08:26.944199 31889 ts_tablet_manager.cc:1397] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20260923 18:08:26.950338 31889 raft_consensus.cc:348] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:26.950747 31889 raft_consensus.cc:374] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260923 18:08:26.950805 31586 tablet.cc:2369] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:08:26.950847 31889 raft_consensus.cc:729] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2923897b74d7459d9b4851212a810f68, State: Initialized, Role: FOLLOWER
I20260923 18:08:26.951403 31889 consensus_queue.cc:260] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:26.952664 31889 ts_tablet_manager.cc:1428] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260923 18:08:26.952885 31852 heartbeater.cc:503] Master 127.18.44.254:37657 was elected leader, sending a full tablet report...
W20260923 18:08:26.986570 31719 tablet.cc:2369] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:08:27.040963 31853 tablet.cc:2369] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:08:27.167462 31891 raft_consensus.cc:482] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:27.167747 31891 raft_consensus.cc:504] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:27.169205 31891 leader_election.cc:288] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8c073e89804662a3d91ff158af4adf (127.18.44.194:35147), 2923897b74d7459d9b4851212a810f68 (127.18.44.195:46651)
I20260923 18:08:27.175645 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.170377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:47461 (local address 127.18.44.194:35147)
0923 18:08:27.170505 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:27.170509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:27.170524 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:08:27.170595 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:27.170598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:27.170645 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:27.170729 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:27.171148 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.172035 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.174174 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.174785 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.174883 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:27.174993 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:27.175384 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 18:08:27.175485 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:08:27.176359 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.169481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35147 (local address 127.18.44.193:47461)
0923 18:08:27.169845 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:27.169864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:27.169997 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:27.170846 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:27.170850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:27.170870 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:27.171012 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:27.171017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.172269 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.172272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:27.174008 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:27.174020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.175094 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.175098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:27.175205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:27.175228 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:08:27.176103 (+ 875us) client_negotiation.cc:240] Negotiation successful
0923 18:08:27.176195 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":151,"threads_started":1}
I20260923 18:08:27.177012 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.170466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:52429 (local address 127.18.44.195:46651)
0923 18:08:27.170593 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:27.170598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:27.170918 (+ 320us) server_negotiation.cc:407] Connection header received
0923 18:08:27.171116 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:27.171120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:27.171169 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:27.171215 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:27.171674 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.172493 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.175494 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.176142 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.176250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:27.176300 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:27.176795 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 18:08:27.176893 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:08:27.177242 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99049260e2274de2a9bc27e4fa9e4e15" candidate_uuid: "4c288621d2a3475c8e8602e25c85a538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8c073e89804662a3d91ff158af4adf" is_pre_election: true
I20260923 18:08:27.177431 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.169944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:46651 (local address 127.18.44.193:52429)
0923 18:08:27.170819 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:27.170833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:27.170995 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:27.171397 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:27.171399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:27.171417 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:27.171541 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:27.171547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.172728 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.172731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:27.175322 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:27.175335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.176413 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.176417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:27.176520 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:27.176545 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:08:27.177197 (+ 652us) client_negotiation.cc:240] Negotiation successful
0923 18:08:27.177277 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":87,"threads_started":1}
I20260923 18:08:27.177596 31670 raft_consensus.cc:2454] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c288621d2a3475c8e8602e25c85a538 in term 0.
I20260923 18:08:27.178012 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99049260e2274de2a9bc27e4fa9e4e15" candidate_uuid: "4c288621d2a3475c8e8602e25c85a538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2923897b74d7459d9b4851212a810f68" is_pre_election: true
I20260923 18:08:27.178287 31806 raft_consensus.cc:2454] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c288621d2a3475c8e8602e25c85a538 in term 0.
I20260923 18:08:27.178833 31471 leader_election.cc:302] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c288621d2a3475c8e8602e25c85a538, ff8c073e89804662a3d91ff158af4adf; no voters:
I20260923 18:08:27.179148 31891 raft_consensus.cc:2781] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:08:27.179245 31891 raft_consensus.cc:482] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:08:27.179317 31891 raft_consensus.cc:3037] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:27.180649 31891 raft_consensus.cc:504] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:27.181035 31891 leader_election.cc:288] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [CANDIDATE]: Term 1 election: Requested vote from peers ff8c073e89804662a3d91ff158af4adf (127.18.44.194:35147), 2923897b74d7459d9b4851212a810f68 (127.18.44.195:46651)
I20260923 18:08:27.181586 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99049260e2274de2a9bc27e4fa9e4e15" candidate_uuid: "4c288621d2a3475c8e8602e25c85a538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2923897b74d7459d9b4851212a810f68"
I20260923 18:08:27.181730 31806 raft_consensus.cc:3037] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:27.181732 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99049260e2274de2a9bc27e4fa9e4e15" candidate_uuid: "4c288621d2a3475c8e8602e25c85a538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8c073e89804662a3d91ff158af4adf"
I20260923 18:08:27.181874 31670 raft_consensus.cc:3037] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:27.183092 31806 raft_consensus.cc:2454] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c288621d2a3475c8e8602e25c85a538 in term 1.
I20260923 18:08:27.183305 31670 raft_consensus.cc:2454] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c288621d2a3475c8e8602e25c85a538 in term 1.
I20260923 18:08:27.183761 31471 leader_election.cc:302] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2923897b74d7459d9b4851212a810f68, 4c288621d2a3475c8e8602e25c85a538; no voters:
I20260923 18:08:27.184000 31891 raft_consensus.cc:2781] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:27.184262 31891 raft_consensus.cc:686] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [term 1 LEADER]: Becoming Leader. State: Replica: 4c288621d2a3475c8e8602e25c85a538, State: Running, Role: LEADER
I20260923 18:08:27.184685 31891 consensus_queue.cc:237] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } }
I20260923 18:08:27.188599 31386 catalog_manager.cc:5495] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c288621d2a3475c8e8602e25c85a538 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "4c288621d2a3475c8e8602e25c85a538" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c288621d2a3475c8e8602e25c85a538" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 } health_report { overall_health: UNKNOWN } } }
I20260923 18:08:27.254330 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.250807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.18.44.193:33901)
0923 18:08:27.250951 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:27.250955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:27.250969 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:08:27.251054 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:27.251057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:27.251111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:27.251203 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:08:27.251477 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.251983 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.252862 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:27.253083 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:27.253179 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:27.253482 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:08:27.253914 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:08:27.254022 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:27.254106 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 18:08:27.254171 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:08:27.260695 31806 raft_consensus.cc:1261] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 1 FOLLOWER]: Refusing update from remote peer 4c288621d2a3475c8e8602e25c85a538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:27.260687 31670 raft_consensus.cc:1261] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 1 FOLLOWER]: Refusing update from remote peer 4c288621d2a3475c8e8602e25c85a538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:27.261436 31891 consensus_queue.cc:1035] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2923897b74d7459d9b4851212a810f68" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:27.261705 31898 consensus_queue.cc:1035] T 99049260e2274de2a9bc27e4fa9e4e15 P 4c288621d2a3475c8e8602e25c85a538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8c073e89804662a3d91ff158af4adf" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:27.268921 31908 mvcc.cc:205] Tried to move back new op lower bound from 7332605572131467264 to 7332605571831488512. Current Snapshot: MvccSnapshot[applied={T|T < 7332605572131467264}]
I20260923 18:08:27.278096 31385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:27.278270 31385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:27.280931 31385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28b1265071da41cbb6143c58c51f0ae9: Sending DeleteTablet for 3 replicas of tablet 99049260e2274de2a9bc27e4fa9e4e15
I20260923 18:08:27.281976 31767 tablet_service.cc:1514] Processing DeleteTablet for tablet 99049260e2274de2a9bc27e4fa9e4e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:27 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:58694
I20260923 18:08:27.282169 31634 tablet_service.cc:1514] Processing DeleteTablet for tablet 99049260e2274de2a9bc27e4fa9e4e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:27 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:59512
I20260923 18:08:27.282536 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31424
I20260923 18:08:27.282593 31917 tablet_replica.cc:330] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: stopping tablet replica
I20260923 18:08:27.282930 31917 raft_consensus.cc:2229] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:27.283334 31917 raft_consensus.cc:2258] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:27.284833 31917 ts_tablet_manager.cc:1905] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:08:27.287567 31918 tablet_replica.cc:330] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: stopping tablet replica
I20260923 18:08:27.287738 31917 ts_tablet_manager.cc:1918] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:27.287842 31917 log.cc:1196] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/99049260e2274de2a9bc27e4fa9e4e15
I20260923 18:08:27.287922 31918 raft_consensus.cc:2229] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:27.288215 31917 ts_tablet_manager.cc:1939] T 99049260e2274de2a9bc27e4fa9e4e15 P 2923897b74d7459d9b4851212a810f68: Deleting consensus metadata
I20260923 18:08:27.288309 31918 raft_consensus.cc:2258] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:27.289606 31370 catalog_manager.cc:4841] TS 2923897b74d7459d9b4851212a810f68 (127.18.44.195:46651): tablet 99049260e2274de2a9bc27e4fa9e4e15 (table test-table [id=25b4d429940940a7b1fe0290476d76ad]) successfully deleted
I20260923 18:08:27.289808 31918 ts_tablet_manager.cc:1905] T 99049260e2274de2a9bc27e4fa9e4e15 P ff8c073e89804662a3d91ff158af4adf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:08:27.290995 31370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:33901 (error 108)
I20260923 18:08:27.291838 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31584
I20260923 18:08:27.292076 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.291772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33901 (local address 127.0.0.1:60374)
0923 18:08:27.291907 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:27.291980 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 18:08:27.292579 31370 catalog_manager.cc:4584] TS 4c288621d2a3475c8e8602e25c85a538 (127.18.44.193:33901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99049260e2274de2a9bc27e4fa9e4e15: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33901: connect: Connection refused (error 111)
I20260923 18:08:27.297042 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31721
I20260923 18:08:27.298105 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:27.297788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:35147 (local address 127.0.0.1:59520)
0923 18:08:27.297933 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:27.297997 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 18:08:27.302408 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31353
I20260923 18:08:27.308502 18611 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 18:08:27.645273 18611 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 18:08:27.645398 18611 mini_ranger.cc:66] Stopping Ranger...
I20260923 18:08:27.993664 18611 mini_ranger.cc:68] Stopped Ranger
2026-09-23 18:08:27.993 UTC [30419] LOG: received smart shutdown request
2026-09-23 18:08:27.999 UTC [30419] LOG: background worker "logical replication launcher" (PID 30428) exited with exit code 1
2026-09-23 18:08:28.000 UTC [30423] LOG: shutting down
2026-09-23 18:08:28.262 UTC [30419] LOG: database system is shut down
2026-09-23T18:08:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57939 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:08:28 dist-test-slave-rvjj krb5kdc[31928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:08:28 dist-test-slave-rvjj krb5kdc[31928](info): set up 2 sockets
Sep 23 18:08:28 dist-test-slave-rvjj krb5kdc[31928](info): commencing operation
krb5kdc: starting...
W20260923 18:08:29.397153 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:08:29 dist-test-slave-rvjj krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:08:29.551357 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:42199
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:29.693310 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:29.693759 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:29.693833 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:29.698750 31941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:29.698882 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:29.698915 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:29.698937 31941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:29.698957 31941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:29.705792 31941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:29.707456 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:29.708904 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:29.720290 31947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:29.720230 31946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:29.720376 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:29.721192 31941 server_base.cc:1034] running on GCE node
I20260923 18:08:29.722024 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:29.723412 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:29.726748 31941 hybrid_clock.cc:648] HybridClock initialized: now 1790186909726684 us; error 89 us; skew 500 ppm
Sep 23 18:08:29 dist-test-slave-rvjj krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:29.730453 31941 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:29.731671 31941 webserver.cc:460] Webserver started at http://127.18.44.254:40273/ using document root <none> and password file <none>
I20260923 18:08:29.732323 31941 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:29.732399 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:29.732622 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:29.734912 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "5965dfb119344957a8604f0d545ae5f8"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "a355b15b4f79e354fe7e10b5e5bc9e49"
server_key_iv: "30b1169d617f04cc1be5bfebfd983ed8"
server_key_version: "encryptionkey@0"
I20260923 18:08:29.735617 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "5965dfb119344957a8604f0d545ae5f8"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "a355b15b4f79e354fe7e10b5e5bc9e49"
server_key_iv: "30b1169d617f04cc1be5bfebfd983ed8"
server_key_version: "encryptionkey@0"
I20260923 18:08:29.740725 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260923 18:08:29.744419 31956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:29.746310 31941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:08:29.746510 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "5965dfb119344957a8604f0d545ae5f8"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "a355b15b4f79e354fe7e10b5e5bc9e49"
server_key_iv: "30b1169d617f04cc1be5bfebfd983ed8"
server_key_version: "encryptionkey@0"
I20260923 18:08:29.746670 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:29.763649 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:29.767544 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:29.767735 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:29.776309 31941 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:42199
I20260923 18:08:29.776360 32008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:42199 every 8 connection(s)
I20260923 18:08:29.777510 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:29.781210 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 31941
I20260923 18:08:29.781339 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:08:29.781629 18611 external_mini_cluster.cc:1428] Setting key 897f9b716553c97ed4543a9fcf96b463
I20260923 18:08:29.782660 32009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:29.790587 32009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.44.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.253@KRBTEST.COM" created.
Sep 23 18:08:29 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:29.827536 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.795029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.18.44.254:42199)
0923 18:08:29.796490 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:29.796508 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:29.796557 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:08:29.799371 (+ 2814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:29.799377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:29.799670 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:29.799868 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:29.801546 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:29.803001 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:29.805283 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:29.806215 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:29.806793 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:29.816539 (+ 9746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:29.816550 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:29.816576 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:29.816611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:29.819860 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:29.821107 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:29.821116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:29.821118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:29.821309 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:29.821808 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:29.821811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:29.821812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:29.823196 (+ 1384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:29.823551 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:29.824734 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0923 18:08:29.825664 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":503,"server-negotiator.queue_time_us":1339,"thread_start_us":388,"threads_started":1}
I20260923 18:08:29.827536 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.794964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42199 (local address 127.0.0.1:44328)
0923 18:08:29.795845 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.795913 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:29.798669 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:29.800087 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:29.800097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:29.800717 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:29.801267 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:29.801300 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:29.803358 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:29.803365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:29.804937 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:29.804968 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:29.806570 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:29.806577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:29.806796 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:29.808040 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:29.808074 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:29.816212 (+ 8138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:29.820138 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:29.820161 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:29.820179 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:29.820722 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:29.821426 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:29.821430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:29.821432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:29.821617 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:29.823632 (+ 2015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:29.823641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:29.823926 (+ 285us) client_negotiation.cc:769] Sending connection context
0923 18:08:29.824329 (+ 403us) client_negotiation.cc:240] Negotiation successful
0923 18:08:29.825454 (+ 1125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":360,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
I20260923 18:08:29.829622 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:41295
--webserver_interface=127.18.44.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:29.832050 32009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:08:29.833274 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.832859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41072)
0923 18:08:29.833040 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.833158 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 18:08:29.833659 31958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
I20260923 18:08:29.834797 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.834521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41088)
0923 18:08:29.834656 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.834716 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 18:08:29.835060 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:41295: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
I20260923 18:08:29.884418 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } attempt: 1
I20260923 18:08:29.886072 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.885660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41098)
0923 18:08:29.885881 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.885963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 18:08:29.887310 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.886987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41104)
0923 18:08:29.887108 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.887172 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 18:08:29.887511 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:41295: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
I20260923 18:08:29.955726 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } attempt: 2
I20260923 18:08:29.956859 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.956511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41110)
0923 18:08:29.956689 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.956761 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 18:08:29.958338 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:29.957946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41114)
0923 18:08:29.958111 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:29.958251 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 18:08:29.958534 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:41295: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
W20260923 18:08:29.963886 32015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:29.964214 32015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:29.964262 32015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:29.968565 32015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:29.968672 32015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:29.968694 32015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:29.968715 32015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:29.968734 32015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:29.974681 32015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:41295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.44.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:29.976085 32015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:29.977268 32015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:29.987771 32021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:29.987684 32022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:29.987879 32015 server_base.cc:1034] running on GCE node
W20260923 18:08:29.987674 32024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:29.988830 32015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:29.990604 32015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:29.991876 32015 hybrid_clock.cc:648] HybridClock initialized: now 1790186909991837 us; error 60 us; skew 500 ppm
Sep 23 18:08:29 dist-test-slave-rvjj krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:29.995913 32015 init.cc:377] Logged in from keytab as kudu/127.18.44.253@KRBTEST.COM (short username kudu)
I20260923 18:08:29.997327 32015 webserver.cc:460] Webserver started at http://127.18.44.253:39901/ using document root <none> and password file <none>
I20260923 18:08:29.998131 32015 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:29.998229 32015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:29.998543 32015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:30.001585 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data/instance:
uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "dc9ec729c60f533917ffd8b4df28450c"
server_key_iv: "9566b87d8430af4133cf8a81ea8aa872"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.002398 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal/instance:
uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "dc9ec729c60f533917ffd8b4df28450c"
server_key_iv: "9566b87d8430af4133cf8a81ea8aa872"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.009274 32015 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260923 18:08:30.013622 32031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:30.016141 32015 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:08:30.016428 32015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f"
format_stamp: "Formatted at 2026-09-23 18:08:29 on dist-test-slave-rvjj"
server_key: "dc9ec729c60f533917ffd8b4df28450c"
server_key_iv: "9566b87d8430af4133cf8a81ea8aa872"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.016618 32015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:30.049189 32015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:30.049826 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } attempt: 3
I20260923 18:08:30.051043 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.050590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41130)
0923 18:08:30.050851 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.050947 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 18:08:30.052034 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.051789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41140)
0923 18:08:30.051905 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.051961 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 18:08:30.052196 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:41295: Network error: Client connection negotiation failed: client connection to 127.18.44.253:41295: connect: Connection refused (error 111)
I20260923 18:08:30.054973 32015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:30.055384 32015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:30.064918 32015 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.253:41295
I20260923 18:08:30.064967 32083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.253:41295 every 8 connection(s)
I20260923 18:08:30.066363 32015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
I20260923 18:08:30.069976 32084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:30.071605 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32015
I20260923 18:08:30.071760 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-1/wal/instance
I20260923 18:08:30.072098 18611 external_mini_cluster.cc:1428] Setting key f6b4ed03ec2579133dd5f29ef5026f26
I20260923 18:08:30.074383 32084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.44.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.252@KRBTEST.COM" created.
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:30.095077 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.077158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.18.44.254:42199)
0923 18:08:30.077405 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.077411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.079676 (+ 2265us) server_negotiation.cc:407] Connection header received
0923 18:08:30.080855 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.080859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.080935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.081105 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.082595 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.083185 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.085028 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.085311 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.085432 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.089784 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.089788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.089790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.089831 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.091643 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.092418 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.092422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.092424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.092513 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.092918 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.092921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.092923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.093161 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.093253 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.094721 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.094889 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 18:08:30.096159 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.078499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42199 (local address 127.0.0.1:44332)
0923 18:08:30.079464 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.079533 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.080537 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.081308 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.081316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.081730 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.082345 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.082362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.083468 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.083474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.084775 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.084801 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.085600 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.085605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.085925 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.087165 (+ 1240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.087209 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.089466 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.091863 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.091872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.091896 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.092298 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.092607 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.092610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.092612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.092819 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.093274 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.093284 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.093657 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.093884 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.094400 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":240,"threads_started":1}
I20260923 18:08:30.098212 32084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
I20260923 18:08:30.103034 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:42495
--webserver_interface=127.18.44.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:08:30.110045 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.098931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41154)
0923 18:08:30.099117 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.099135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.099295 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.099951 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.099955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.100129 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.100268 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.100275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.101529 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.101533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.102324 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.102336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.103461 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.103464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.103571 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.104178 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.104191 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.105844 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.108248 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.108251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.108254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.108481 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.108822 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.108825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.108827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.108910 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.109452 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.109455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.109545 (+ 90us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.109641 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.109825 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 18:08:30.110062 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.098986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.18.44.253:41295)
0923 18:08:30.099465 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.099476 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.099500 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:08:30.099552 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.099558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.099761 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.099926 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.100385 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.101395 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.102474 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.103303 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.103446 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.106037 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.106052 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.106074 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.106102 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.108104 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.108598 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.108609 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.108611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.108726 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.108988 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.108992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.108994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.109347 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.109490 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.109760 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.109916 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260923 18:08:30.113072 32084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:08:30.114099 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.113752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49262)
0923 18:08:30.113901 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.114003 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 18:08:30.114431 32035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
I20260923 18:08:30.115490 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.115205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49264)
0923 18:08:30.115359 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.115414 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 18:08:30.115629 32084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:42495: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
I20260923 18:08:30.165010 32084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } attempt: 1
I20260923 18:08:30.166148 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.165796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49276)
0923 18:08:30.165971 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.166043 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 18:08:30.166909 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.166674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49286)
0923 18:08:30.166781 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.166838 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 18:08:30.167102 32084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:42495: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
I20260923 18:08:30.195343 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } attempt: 4
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:08:30.207384 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.196118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41170)
0923 18:08:30.196421 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.196443 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.196602 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.196930 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.196934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.197159 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.197317 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.197325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.198655 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.198660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.199576 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.199589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.200269 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.200273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.200399 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.201004 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.201018 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.202799 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.205161 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.205166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.205168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.205506 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.205989 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.205993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.205995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.206108 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.206653 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.206657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.206814 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.206944 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.207136 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260923 18:08:30.207520 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.196252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.18.44.253:41295)
0923 18:08:30.196465 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.196470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.196543 (+ 73us) server_negotiation.cc:407] Connection header received
0923 18:08:30.196733 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.196736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.196793 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.196867 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.197469 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.198508 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.199750 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.200110 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.200236 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.203144 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.203150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.203153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.203198 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.204959 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.205698 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.205704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.205706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.205843 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.206285 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.206289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.206290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.206543 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.206634 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.207061 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.207221 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 18:08:30.209237 32009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:08:30.210475 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.210039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49288)
0923 18:08:30.210290 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.210371 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 18:08:30.211479 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.211089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49296)
0923 18:08:30.211324 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.211401 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 18:08:30.211637 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:42495: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
W20260923 18:08:30.225399 32090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:30.225719 32090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:30.225764 32090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:30.230110 32090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:30.230204 32090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:30.230223 32090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:30.230239 32090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:30.230253 32090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:30.235440 32084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } attempt: 2
I20260923 18:08:30.236270 32090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:42199,127.18.44.253:41295,127.18.44.252:42495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:42495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.44.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:30.236649 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.236082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49308)
0923 18:08:30.236369 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.236497 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260923 18:08:30.237521 32090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:30.237564 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.237318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49318)
0923 18:08:30.237435 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.237489 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 18:08:30.237778 32084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:42495: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
I20260923 18:08:30.238696 32090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:30.249423 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:30.249490 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:30.249688 32090 server_base.cc:1034] running on GCE node
W20260923 18:08:30.249496 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:30.250517 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:30.251950 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:30.255365 32090 hybrid_clock.cc:648] HybridClock initialized: now 1790186910255292 us; error 82 us; skew 500 ppm
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186910, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:30.258862 32090 init.cc:377] Logged in from keytab as kudu/127.18.44.252@KRBTEST.COM (short username kudu)
I20260923 18:08:30.260011 32090 webserver.cc:460] Webserver started at http://127.18.44.252:34533/ using document root <none> and password file <none>
I20260923 18:08:30.260663 32090 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:30.260742 32090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:30.260951 32090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:30.263305 32090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data/instance:
uuid: "3d8caf7716e2414f980cbffe1a752367"
format_stamp: "Formatted at 2026-09-23 18:08:30 on dist-test-slave-rvjj"
server_key: "70fe89f3dcb2214cdc902cda738ca7d8"
server_key_iv: "f218122bb53c6759ce730f4c2df10b74"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.263963 32090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal/instance:
uuid: "3d8caf7716e2414f980cbffe1a752367"
format_stamp: "Formatted at 2026-09-23 18:08:30 on dist-test-slave-rvjj"
server_key: "70fe89f3dcb2214cdc902cda738ca7d8"
server_key_iv: "f218122bb53c6759ce730f4c2df10b74"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.268479 32090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:08:30.270833 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } attempt: 1
I20260923 18:08:30.271936 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.271500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49334)
0923 18:08:30.271706 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.271809 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 18:08:30.271914 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:30.273015 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.272653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49346)
0923 18:08:30.272842 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.272924 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260923 18:08:30.273259 32009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:42495: Network error: Client connection negotiation failed: client connection to 127.18.44.252:42495: connect: Connection refused (error 111)
I20260923 18:08:30.273887 32090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:08:30.274049 32090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "3d8caf7716e2414f980cbffe1a752367"
format_stamp: "Formatted at 2026-09-23 18:08:30 on dist-test-slave-rvjj"
server_key: "70fe89f3dcb2214cdc902cda738ca7d8"
server_key_iv: "f218122bb53c6759ce730f4c2df10b74"
server_key_version: "encryptionkey@0"
I20260923 18:08:30.274175 32090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:30.303879 32090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:30.307948 32090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:30.308148 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:30.316696 32090 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.252:42495
I20260923 18:08:30.316932 32158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.252:42495 every 8 connection(s)
I20260923 18:08:30.317813 32090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
I20260923 18:08:30.322147 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:30.323081 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32090
I20260923 18:08:30.323284 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790186727112038-18611-0/minicluster-data/master-2/wal/instance
I20260923 18:08:30.323684 18611 external_mini_cluster.cc:1428] Setting key 5ad4a3d9f6980b66f6ba06f059a68df2
I20260923 18:08:30.329049 32084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } attempt: 3
I20260923 18:08:30.330408 32159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:08:30.340436 32009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } attempt: 2
I20260923 18:08:30.341269 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.329708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49366)
0923 18:08:30.329881 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.329899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.330009 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.330545 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.330550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.330822 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.330990 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.330999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.332189 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.332193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.333062 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.333076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.333809 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.333813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.333908 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.334706 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.334723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.336407 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.339035 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.339039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.339041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.339335 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.339726 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.339730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.339732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.339833 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.340658 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.340661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.340795 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.340883 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.341065 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 18:08:30.343245 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.329806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.18.44.252:42495)
0923 18:08:30.330199 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.330209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.330223 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:08:30.330260 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.330263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.330308 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.330404 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.331171 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.332019 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.333262 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.333638 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.334225 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.336585 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.336589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.336592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.336613 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.338876 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.339450 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.339455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.339457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.339603 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.339949 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.339951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.339952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.340523 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.341620 (+ 1097us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.341686 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.342031 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":319,"thread_start_us":192,"threads_started":1}
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186910, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:30.343248 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.325441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.18.44.252:42495)
0923 18:08:30.325983 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.326001 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.326045 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:08:30.326981 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.327009 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.327546 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.327922 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.329241 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.330805 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.332194 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.332793 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.332962 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.336167 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.336175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.336189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.336228 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.338628 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.339187 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.339197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.339207 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.339637 (+ 430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.340124 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.340128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.340130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.340545 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.340746 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.341641 (+ 895us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.342122 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":375,"thread_start_us":175,"threads_started":1}
I20260923 18:08:30.348587 32084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: Bootstrap starting.
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:08:30.352137 32084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:30.352766 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.334548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.18.44.254:42199)
0923 18:08:30.334716 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.334721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.334737 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:08:30.334793 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.334796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.334852 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.334971 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.336055 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.336715 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.338043 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.338616 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.338775 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.345868 (+ 7093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.345872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.345875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.345911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.349070 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.350197 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.350201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.350203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.350350 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.350692 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.350696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.350698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.351063 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.351157 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.351977 (+ 820us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.352501 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":13184}
I20260923 18:08:30.353220 32084 log.cc:824] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:30.353513 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.333460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42199 (local address 127.0.0.1:44346)
0923 18:08:30.334045 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.334135 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.334569 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.335088 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.335096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.335673 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.335881 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.335890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.336865 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.336868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.337880 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.337894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.339039 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.339044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.339201 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.340274 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.340290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.345638 (+ 5348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.349464 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.349491 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.349501 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.350042 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.350447 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.350450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.350453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.350566 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.351441 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.351451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.351647 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.352137 (+ 490us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.352682 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":169,"threads_started":1}
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:30.355564 32159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:08:30.356326 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.341261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49380)
0923 18:08:30.341654 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.341669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.341760 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.342720 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.342723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.342948 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.343151 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.343160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.344532 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.344537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.345621 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.345640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.346607 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.346617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.346731 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.347669 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.347688 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.349397 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.351900 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.351904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.351906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.352180 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.353320 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.353324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.353326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.353446 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.355641 (+ 2195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.355645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.355792 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.355934 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.356131 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260923 18:08:30.357398 32084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: No bootstrap required, opened a new log
I20260923 18:08:30.357702 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.341916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.18.44.252:42495)
0923 18:08:30.342402 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.342414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.342429 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:08:30.342473 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.342477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.342548 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.343380 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.343543 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.344367 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.345823 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.346301 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.346605 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.349586 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.349590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.349593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.349620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.351772 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.353018 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.353022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.353024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.353175 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.353562 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.353566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.353568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.353847 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.355644 (+ 1797us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.357248 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.357491 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":152,"threads_started":1}
I20260923 18:08:30.361892 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.346802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.18.44.254:42199)
0923 18:08:30.347600 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.347623 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.347641 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:08:30.347718 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.347721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.347776 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.347922 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.348810 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.349705 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.351414 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.351792 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.351898 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.354840 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.354845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.354847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.354879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.359225 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.359859 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.359863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.359865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.359970 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.360286 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.360289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.360291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.360537 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.360777 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.361503 (+ 726us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.361649 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"spinlock_wait_cycles":2304,"thread_start_us":244,"threads_started":1}
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186910, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:08:30.362378 32009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: Bootstrap starting.
I20260923 18:08:30.362769 32084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.363121 32084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:30.363202 32084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1cf262935b4afb9c2a7fb5fd287d4f, State: Initialized, Role: FOLLOWER
I20260923 18:08:30.363924 32084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.364751 32168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.364988 32168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.365926 32084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:30.366137 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.356665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.18.44.253:41295)
0923 18:08:30.356811 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.356815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.356832 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:08:30.356880 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.356883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.356954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.357043 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.357649 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.358345 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.359465 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.359886 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.359983 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.362666 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.362669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.362671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.362724 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.364280 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.364774 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.364777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.364779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.364867 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.365189 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.365193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.365194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.365449 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.365543 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.365820 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.365930 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:08:30.366155 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.356308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:41295 (local address 127.0.0.1:41172)
0923 18:08:30.356664 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.356681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.356783 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.357203 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.357206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.357373 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.357512 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.357519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.358456 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.358459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.359335 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.359346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.360124 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.360128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.360239 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.360771 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.360786 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.362477 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.364399 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.364402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.364405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.364653 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.364978 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.364981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.364983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.365108 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.365555 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.365558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.365657 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.365746 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.365912 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260923 18:08:30.367319 32159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:08:30.366410 32009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:30.369802 32009 log.cc:824] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: Log is configured to *not* fsync() on all Append() calls
W20260923 18:08:30.372896 32179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:08:30.373407 32179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186910, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:08:30.371191 32015 master_runner.cc:186] Error getting master registration for 127.18.44.254:42199: OK, instance_id { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" instance_seqno: 1790186909773880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:08:30.375545 32015 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:08:30.380010 32009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: No bootstrap required, opened a new log
I20260923 18:08:30.380236 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.367885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:42495 (local address 127.0.0.1:49386)
0923 18:08:30.368218 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:30.368233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:30.368321 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:30.369650 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:30.369653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:30.369861 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:30.369984 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.369992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.371065 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.371068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.371956 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:30.371968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.372596 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.372599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:30.372685 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.373361 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:30.373376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:30.375319 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:30.378509 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.378513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.378516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.378791 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.379072 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:30.379075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:30.379077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:30.379174 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:30.379668 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:30.379671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:30.379763 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 18:08:30.379832 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:08:30.380001 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260923 18:08:30.381067 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.368102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.18.44.252:42495)
0923 18:08:30.369362 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.369367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.369383 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:08:30.369430 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.369433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.369492 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.369589 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.370112 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.370945 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.372108 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.372475 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.373018 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.375528 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.375532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.375534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.375563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.378370 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.378901 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.378905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.378907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.378986 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.379307 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.379310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.379311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.379560 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.380682 (+ 1122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.380771 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.380915 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150}
Sep 23 18:08:30 dist-test-slave-rvjj krb5kdc[31928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:08:30.385732 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: Bootstrap starting.
I20260923 18:08:30.387180 32009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.387733 32009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:30.387771 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.368585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.18.44.253:41295)
0923 18:08:30.368735 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.368740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.368955 (+ 215us) server_negotiation.cc:407] Connection header received
0923 18:08:30.369106 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.369109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.369156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.369252 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.376642 (+ 7390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.377428 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.378660 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.378998 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.379087 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.382684 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.382689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.382692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.382738 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.385131 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.385810 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.385815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.385817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.385931 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.386279 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.386284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.386285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.386578 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.386888 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.387369 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.387563 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:08:30.387998 32009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5965dfb119344957a8604f0d545ae5f8, State: Initialized, Role: FOLLOWER
I20260923 18:08:30.388736 32009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.390303 32159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:30.391579 32159 log.cc:824] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:30.391824 32009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:30.393334 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:08:30.394904 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: No bootstrap required, opened a new log
I20260923 18:08:30.399720 32182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.399973 32182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.400293 32159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.400633 32159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:30.400694 32159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8caf7716e2414f980cbffe1a752367, State: Initialized, Role: FOLLOWER
I20260923 18:08:30.401405 32159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.403779 32159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:30.402626 31941 master_runner.cc:186] Error getting master registration for 127.18.44.252:42495: OK, instance_id { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" instance_seqno: 1790186910314366 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260923 18:08:30.407752 32194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:08:30.407869 32194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:08:30.408799 31941 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:08:30.416544 32090 master_runner.cc:263] No leader master found from master 3d8caf7716e2414f980cbffe1a752367
I20260923 18:08:30.416705 32090 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:08:30.422801 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.401333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.18.44.254:42199)
0923 18:08:30.401477 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.401482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.408481 (+ 6999us) server_negotiation.cc:407] Connection header received
0923 18:08:30.408667 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.408671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.408725 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.408778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.410705 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.411554 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.413064 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.413622 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.414156 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.417600 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.417606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.417609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.417657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.420432 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.421085 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.421089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.421091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.421200 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.421513 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.421516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.421518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.421803 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.421959 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.422461 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.422632 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 18:08:30.425247 32214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:08:30.425396 32214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:08:30.426364 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.400820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.18.44.252:42495)
0923 18:08:30.400964 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.400968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.403601 (+ 2633us) server_negotiation.cc:407] Connection header received
0923 18:08:30.406837 (+ 3236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.406843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.406914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.406980 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.408206 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.408972 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.414430 (+ 5458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.414880 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.414983 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.417538 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.417544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.417547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.417601 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.420428 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.424636 (+ 4208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.424641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.424643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.424768 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.425081 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.425084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.425086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.425343 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.425501 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.426032 (+ 531us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.426181 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:08:30.427627 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:30.402132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.18.44.253:41295)
0923 18:08:30.402260 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:30.402265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:30.410975 (+ 8710us) server_negotiation.cc:407] Connection header received
0923 18:08:30.411191 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:30.411195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:30.411305 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:30.411364 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:30.412403 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.413623 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.418101 (+ 4478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:30.418469 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:30.418596 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:30.422584 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:30.422589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:30.422592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:30.422637 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:30.425003 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.425673 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.425678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.425680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.425797 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:30.426131 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:30.426135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:30.426137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:30.426430 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:30.426551 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:30.427274 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 18:08:30.427448 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:08:30.430872 32182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:30.431125 32182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.432736 32182 leader_election.cc:288] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c1cf262935b4afb9c2a7fb5fd287d4f (127.18.44.253:41295), 3d8caf7716e2414f980cbffe1a752367 (127.18.44.252:42495)
I20260923 18:08:30.433918 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5965dfb119344957a8604f0d545ae5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8caf7716e2414f980cbffe1a752367" is_pre_election: true
I20260923 18:08:30.434459 32133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5965dfb119344957a8604f0d545ae5f8 in term 0.
I20260923 18:08:30.435098 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5965dfb119344957a8604f0d545ae5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" is_pre_election: true
I20260923 18:08:30.435539 32057 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5965dfb119344957a8604f0d545ae5f8 in term 0.
I20260923 18:08:30.435748 31960 leader_election.cc:302] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8caf7716e2414f980cbffe1a752367, 5965dfb119344957a8604f0d545ae5f8; no voters:
I20260923 18:08:30.436896 32182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:08:30.437021 32182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:08:30.437068 32182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:30.438457 32182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.439426 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5965dfb119344957a8604f0d545ae5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8caf7716e2414f980cbffe1a752367"
I20260923 18:08:30.439504 32057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5965dfb119344957a8604f0d545ae5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f"
I20260923 18:08:30.439612 32133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:30.439653 32057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:30.441350 32133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5965dfb119344957a8604f0d545ae5f8 in term 1.
I20260923 18:08:30.441350 32057 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5965dfb119344957a8604f0d545ae5f8 in term 1.
I20260923 18:08:30.441879 31958 leader_election.cc:302] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d8caf7716e2414f980cbffe1a752367, 5965dfb119344957a8604f0d545ae5f8, 6c1cf262935b4afb9c2a7fb5fd287d4f; no voters:
I20260923 18:08:30.441946 32182 leader_election.cc:288] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [CANDIDATE]: Term 1 election: Requested vote from peers 6c1cf262935b4afb9c2a7fb5fd287d4f (127.18.44.253:41295), 3d8caf7716e2414f980cbffe1a752367 (127.18.44.252:42495)
I20260923 18:08:30.442358 32182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:30.442276 32200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.442440 32200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.442680 32182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5965dfb119344957a8604f0d545ae5f8, State: Running, Role: LEADER
I20260923 18:08:30.448452 32182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } }
I20260923 18:08:30.453169 32182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5965dfb119344957a8604f0d545ae5f8. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.453413 32182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:30.457903 32218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:30.462821 32218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:30.476564 32133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [term 1 FOLLOWER]: Refusing update from remote peer 5965dfb119344957a8604f0d545ae5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:30.476718 32057 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [term 1 FOLLOWER]: Refusing update from remote peer 5965dfb119344957a8604f0d545ae5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:30.477558 32182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:30.478636 32216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:30.484027 32168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5965dfb119344957a8604f0d545ae5f8. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.484216 32168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.490012 32200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5965dfb119344957a8604f0d545ae5f8. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.490201 32200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.497947 32216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.498142 32216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:30.498268 32219 mvcc.cc:205] Tried to move back new op lower bound from 7332605585285967872 to 7332605585202675712. Current Snapshot: MvccSnapshot[applied={T|T < 7332605585285967872}]
I20260923 18:08:30.499934 32168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.500104 32168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.509331 32218 catalog_manager.cc:1347] Generated new cluster ID: 2d871cc2f2824694a810c3901e69bca8
I20260923 18:08:30.509443 32218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:30.509814 32182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.509969 32182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:30.511703 32200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5965dfb119344957a8604f0d545ae5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5965dfb119344957a8604f0d545ae5f8" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42199 } } peers { permanent_uuid: "6c1cf262935b4afb9c2a7fb5fd287d4f" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 41295 } } peers { permanent_uuid: "3d8caf7716e2414f980cbffe1a752367" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 42495 } } }
I20260923 18:08:30.511881 32200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:08:30.512799 32222 mvcc.cc:205] Tried to move back new op lower bound from 7332605585285967872 to 7332605585202675712. Current Snapshot: MvccSnapshot[applied={T|T < 7332605585285967872}]
I20260923 18:08:30.531683 32218 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:08:30.533150 32218 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:30.546800 32218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5965dfb119344957a8604f0d545ae5f8: Generated new TSK 0
I20260923 18:08:30.548044 32218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:08:31.374876 32179 catalog_manager.cc:1259] Loaded cluster ID: 2d871cc2f2824694a810c3901e69bca8
I20260923 18:08:31.374977 32179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: loading cluster ID for follower catalog manager: success
I20260923 18:08:31.376883 32179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: acquiring CA information for follower catalog manager: success
I20260923 18:08:31.378285 32179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c1cf262935b4afb9c2a7fb5fd287d4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:08:31.426759 32214 catalog_manager.cc:1259] Loaded cluster ID: 2d871cc2f2824694a810c3901e69bca8
I20260923 18:08:31.426870 32214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: loading cluster ID for follower catalog manager: success
I20260923 18:08:31.428737 32214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: acquiring CA information for follower catalog manager: success
I20260923 18:08:31.430322 32214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d8caf7716e2414f980cbffe1a752367: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:08:31.627507 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 31941
I20260923 18:08:31.635747 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32015
I20260923 18:08:31.642633 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32090
2026-09-23T18:08:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3269 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:08:31 dist-test-slave-rvjj krb5kdc[32234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:08:31 dist-test-slave-rvjj krb5kdc[32234](info): set up 2 sockets
Sep 23 18:08:31 dist-test-slave-rvjj krb5kdc[32234](info): commencing operation
krb5kdc: starting...
W20260923 18:08:32.661597 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:08:32 dist-test-slave-rvjj krb5kdc[32234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:08:32.819955 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36359
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:32.976877 32247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:32.977337 32247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:32.977389 32247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:32.984678 32247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:32.984830 32247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:32.984855 32247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:32.984875 32247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:32.984894 32247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:32.997260 32247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:36359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:33.000703 32247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:33.002393 32247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:33.015115 32255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:33.015131 32253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:33.015195 32247 server_base.cc:1034] running on GCE node
W20260923 18:08:33.015131 32252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:33.016582 32247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:33.018414 32247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:33.019835 32247 hybrid_clock.cc:648] HybridClock initialized: now 1790186913019771 us; error 77 us; skew 500 ppm
Sep 23 18:08:33 dist-test-slave-rvjj krb5kdc[32234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186913, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:33.024398 32247 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:33.026145 32247 webserver.cc:460] Webserver started at http://127.18.44.254:40509/ using document root <none> and password file <none>
I20260923 18:08:33.026983 32247 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:33.027078 32247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:33.027396 32247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:33.031051 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "7cd993479f404e61bba943e2f87502df"
format_stamp: "Formatted at 2026-09-23 18:08:33 on dist-test-slave-rvjj"
server_key: "e105b24b10bd5acec722f6c679877eaa"
server_key_iv: "420a3b8bd9b3c790925c666aa6b4bd49"
server_key_version: "encryptionkey@0"
I20260923 18:08:33.032037 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "7cd993479f404e61bba943e2f87502df"
format_stamp: "Formatted at 2026-09-23 18:08:33 on dist-test-slave-rvjj"
server_key: "e105b24b10bd5acec722f6c679877eaa"
server_key_iv: "420a3b8bd9b3c790925c666aa6b4bd49"
server_key_version: "encryptionkey@0"
I20260923 18:08:33.039708 32247 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 18:08:33.044631 32262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:33.047039 32247 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:08:33.047240 32247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "7cd993479f404e61bba943e2f87502df"
format_stamp: "Formatted at 2026-09-23 18:08:33 on dist-test-slave-rvjj"
server_key: "e105b24b10bd5acec722f6c679877eaa"
server_key_iv: "420a3b8bd9b3c790925c666aa6b4bd49"
server_key_version: "encryptionkey@0"
I20260923 18:08:33.047454 32247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:33.068503 32247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:33.074066 32247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:33.074433 32247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:33.085541 32247 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:36359
I20260923 18:08:33.085608 32314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:36359 every 8 connection(s)
I20260923 18:08:33.087086 32247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:33.089983 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32247
I20260923 18:08:33.090129 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:08:33.090437 18611 external_mini_cluster.cc:1428] Setting key cb2f98613a9770e4ed08dcec53ad5480
I20260923 18:08:33.091311 32315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:33.099213 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df: Bootstrap starting.
I20260923 18:08:33.101954 32315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:33.102802 32315 log.cc:824] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:08:33 dist-test-slave-rvjj krb5kdc[32234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:33.105666 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df: No bootstrap required, opened a new log
I20260923 18:08:33.110064 32315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } }
I20260923 18:08:33.110391 32315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:33.110453 32315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd993479f404e61bba943e2f87502df, State: Initialized, Role: FOLLOWER
I20260923 18:08:33.110967 32315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } }
I20260923 18:08:33.111125 32315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:33.111202 32315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:33.111346 32315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:33.112061 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:33.092691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.18.44.254:36359)
0923 18:08:33.093489 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:33.093519 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:33.093585 (+ 66us) server_negotiation.cc:407] Connection header received
0923 18:08:33.094784 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:33.094821 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:33.095395 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:33.095832 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:33.097368 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:33.099011 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:33.100355 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:33.101068 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:33.101289 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:33.104655 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:33.104664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:33.104689 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:33.104734 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:33.107122 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:33.107964 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:33.107971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:33.107979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:33.108186 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:33.108572 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:33.108576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:33.108577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:33.109272 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:33.109524 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:33.110161 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 18:08:33.110701 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":212,"threads_started":1}
I20260923 18:08:33.112474 32315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } }
I20260923 18:08:33.112895 32315 leader_election.cc:302] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cd993479f404e61bba943e2f87502df; no voters:
I20260923 18:08:33.113310 32315 leader_election.cc:288] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:08:33.113458 32320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:33.113922 32320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [term 1 LEADER]: Becoming Leader. State: Replica: 7cd993479f404e61bba943e2f87502df, State: Running, Role: LEADER
I20260923 18:08:33.114553 32320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } }
I20260923 18:08:33.115206 32315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:33.117272 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:08:33.117609 32321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cd993479f404e61bba943e2f87502df. Latest consensus state: current_term: 1 leader_uuid: "7cd993479f404e61bba943e2f87502df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } } }
I20260923 18:08:33.117836 32321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:33.118499 32322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cd993479f404e61bba943e2f87502df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd993479f404e61bba943e2f87502df" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36359 } } }
I20260923 18:08:33.119150 32322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:33.119539 32334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:33.125038 32334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:33.133208 32334 catalog_manager.cc:1347] Generated new cluster ID: 18711140a1244d52b2fa0b5ffa62711e
I20260923 18:08:33.133358 32334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:33.149626 32334 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:08:33.151647 32334 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:33.165922 32334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cd993479f404e61bba943e2f87502df: Generated new TSK 0
I20260923 18:08:33.167065 32334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:08:33.178714 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32247
2026-09-23T18:08:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1534 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:08:33 dist-test-slave-rvjj krb5kdc[32342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:08:33 dist-test-slave-rvjj krb5kdc[32342](info): set up 2 sockets
Sep 23 18:08:33 dist-test-slave-rvjj krb5kdc[32342](info): commencing operation
krb5kdc: starting...
W20260923 18:08:34.201071 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:08:34 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:08:34.378990 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:34.593349 32355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:34.593838 32355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:34.593927 32355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:34.598765 32355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:34.599000 32355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:34.599063 32355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:34.599098 32355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:34.599124 32355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:34.606195 32355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:34.607506 32355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:34.608682 32355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:34.620759 32361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:34.620795 32363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:34.621058 32355 server_base.cc:1034] running on GCE node
W20260923 18:08:34.620865 32360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:34.622284 32355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:34.624092 32355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:34.627528 32355 hybrid_clock.cc:648] HybridClock initialized: now 1790186914627451 us; error 65 us; skew 500 ppm
Sep 23 18:08:34 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186914, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:34.633625 32355 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:34.634882 32355 webserver.cc:460] Webserver started at http://127.18.44.254:39025/ using document root <none> and password file <none>
I20260923 18:08:34.636287 32355 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:34.636472 32355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:34.636816 32355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:34.641367 32355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "080cf4f857144262a8f16c4677a1ba20"
format_stamp: "Formatted at 2026-09-23 18:08:34 on dist-test-slave-rvjj"
server_key: "abc6f1533d44c3232db6ca8620da6086"
server_key_iv: "66033d93a83504820604311d98b82ae1"
server_key_version: "encryptionkey@0"
I20260923 18:08:34.642310 32355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "080cf4f857144262a8f16c4677a1ba20"
format_stamp: "Formatted at 2026-09-23 18:08:34 on dist-test-slave-rvjj"
server_key: "abc6f1533d44c3232db6ca8620da6086"
server_key_iv: "66033d93a83504820604311d98b82ae1"
server_key_version: "encryptionkey@0"
I20260923 18:08:34.650753 32355 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260923 18:08:34.656358 32370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:34.658514 32355 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260923 18:08:34.658887 32355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "080cf4f857144262a8f16c4677a1ba20"
format_stamp: "Formatted at 2026-09-23 18:08:34 on dist-test-slave-rvjj"
server_key: "abc6f1533d44c3232db6ca8620da6086"
server_key_iv: "66033d93a83504820604311d98b82ae1"
server_key_version: "encryptionkey@0"
I20260923 18:08:34.659390 32355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:34.721582 32355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:34.728650 32355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:34.729604 32355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:34.742208 32355 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45539
I20260923 18:08:34.742390 32422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45539 every 8 connection(s)
I20260923 18:08:34.743901 32355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:34.747646 32423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:34.754477 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32355
I20260923 18:08:34.754638 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:08:34.754645 32423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap starting.
I20260923 18:08:34.755046 18611 external_mini_cluster.cc:1428] Setting key 81ecdb79176ee909079ce0ac0af04aac
I20260923 18:08:34.757840 32423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:34.758705 32423 log.cc:824] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:34.762357 32423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: No bootstrap required, opened a new log
I20260923 18:08:34.766855 32423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:34.767148 32423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:34.767213 32423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Initialized, Role: FOLLOWER
I20260923 18:08:34.767712 32423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:34.767884 32423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:34.767959 32423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:34.768079 32423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:34.769214 32423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:34.769605 32423 leader_election.cc:302] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080cf4f857144262a8f16c4677a1ba20; no voters:
I20260923 18:08:34.770021 32423 leader_election.cc:288] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:08:34.770421 32428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:34.770905 32428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 LEADER]: Becoming Leader. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Running, Role: LEADER
I20260923 18:08:34.771543 32423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:34.771672 32428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
Sep 23 18:08:34 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:34.774283 32429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:34.774503 32429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:34.774987 32443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:34.775240 32430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080cf4f857144262a8f16c4677a1ba20. Latest consensus state: current_term: 1 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:34.776036 32430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:34.780635 32443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:34.784134 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:34.757015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.18.44.254:45539)
0923 18:08:34.757647 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:34.757668 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:34.757721 (+ 53us) server_negotiation.cc:407] Connection header received
0923 18:08:34.758720 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:34.758881 (+ 161us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:34.760140 (+ 1259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:34.760629 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:34.761755 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:34.762979 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:34.764317 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:34.764879 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:34.765039 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:34.774298 (+ 9259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:34.774307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:34.774336 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:34.774389 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:34.778122 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:34.779318 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:34.779328 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:34.779337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:34.779601 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:34.780057 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:34.780060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:34.780062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:34.780812 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:34.781121 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:34.781528 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 18:08:34.782329 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":198,"threads_started":1}
I20260923 18:08:34.791463 32443 catalog_manager.cc:1347] Generated new cluster ID: 710008fc681c448f8ab61b528aa2dcb8
I20260923 18:08:34.791563 32443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:34.807688 32443 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:08:34.809088 32443 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:34.816488 32443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Generated new TSK 0
I20260923 18:08:34.817567 32443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:08:34.872828 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:35.017102 32449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:35.017452 32449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:35.017499 32449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:35.024700 32449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:35.024820 32449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:35.024891 32449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:35.032384 32449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:35.033694 32449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:35.034811 32449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:35.048499 32455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.048477 32457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.048617 32454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:35.048893 32449 server_base.cc:1034] running on GCE node
I20260923 18:08:35.050279 32449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:35.051865 32449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:35.053155 32449 hybrid_clock.cc:648] HybridClock initialized: now 1790186915053121 us; error 72 us; skew 500 ppm
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:35.058492 32449 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:35.060165 32449 webserver.cc:460] Webserver started at http://127.18.44.193:46263/ using document root <none> and password file <none>
I20260923 18:08:35.061342 32449 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:35.061491 32449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:35.061774 32449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:35.065387 32449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "4e2b445b086e4ae79d0f7980251595db"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "c3a32a6368277024106b3905f2ac20a2"
server_key_iv: "6bbd4a1ce44cb4583ff70bb8450cfd93"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.066893 32449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "4e2b445b086e4ae79d0f7980251595db"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "c3a32a6368277024106b3905f2ac20a2"
server_key_iv: "6bbd4a1ce44cb4583ff70bb8450cfd93"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.075814 32449 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260923 18:08:35.081084 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.083518 32449 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:08:35.083811 32449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "4e2b445b086e4ae79d0f7980251595db"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "c3a32a6368277024106b3905f2ac20a2"
server_key_iv: "6bbd4a1ce44cb4583ff70bb8450cfd93"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.084012 32449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:35.105122 32449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:35.113377 32449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:35.113682 32449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:35.114843 32449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:35.116982 32449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:35.117425 32449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.117702 32449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:35.117841 32449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.136585 32449 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:41739
I20260923 18:08:35.136607 32577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:41739 every 8 connection(s)
I20260923 18:08:35.138357 32449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:35.144672 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32449
I20260923 18:08:35.144788 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:08:35.145565 18611 external_mini_cluster.cc:1428] Setting key e9890049420d5a0e3a41132fd8860a88
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:08:35.161758 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.141748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:37505 (local address 127.18.44.254:45539)
0923 18:08:35.142079 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.142084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.143464 (+ 1380us) server_negotiation.cc:407] Connection header received
0923 18:08:35.145599 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.145603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.145686 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.145789 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:35.148162 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.148916 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.150831 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.151400 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.151539 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.156424 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:35.156431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:35.156435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:35.156507 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:35.158849 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.159835 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.159840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.159841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.159958 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.160383 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.160386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.160388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.160629 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:35.160734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.161378 (+ 644us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.161569 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260923 18:08:35.163020 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.142249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:37505)
0923 18:08:35.143142 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.143223 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.145206 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.146266 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.146276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.146890 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:35.147980 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.148007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.149105 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.149109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.150645 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.150662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.151848 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.151854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.152129 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.153106 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:35.153141 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:35.155919 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:35.159118 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.159127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.159153 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.159668 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.160085 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.160087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.160089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.160253 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.160765 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:35.160773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:35.161095 (+ 322us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.161288 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.161700 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":270,"threads_started":1}
I20260923 18:08:35.165733 32578 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:35.166121 32578 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:08:35.166911 32578 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:35.170274 32383 ts_manager.cc:194] Registered new tserver with Master: 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739)
I20260923 18:08:35.173352 32383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:37505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:08:35.192835 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:35.380719 32583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:35.381358 32583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:35.381500 32583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:35.387481 32583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:35.387616 32583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:35.387691 32583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:35.400147 32583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:35.403110 32583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:35.405238 32583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:35.424432 32589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.425376 32591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.425395 32588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:35.425813 32583 server_base.cc:1034] running on GCE node
I20260923 18:08:35.426668 32583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:35.427512 32583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:35.430974 32583 hybrid_clock.cc:648] HybridClock initialized: now 1790186915430691 us; error 298 us; skew 500 ppm
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:35.435352 32583 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:35.436590 32583 webserver.cc:460] Webserver started at http://127.18.44.194:43105/ using document root <none> and password file <none>
I20260923 18:08:35.437301 32583 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:35.437387 32583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:35.437608 32583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:35.440210 32583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "d36fde91c2d4471f8642db21358dccff"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "24e32b365902d3040cadae28257ebc1e"
server_key_iv: "82ff9a81d69b303d5c1266284ee0094d"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.440939 32583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "d36fde91c2d4471f8642db21358dccff"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "24e32b365902d3040cadae28257ebc1e"
server_key_iv: "82ff9a81d69b303d5c1266284ee0094d"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.447726 32583 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 18:08:35.451694 32598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.454836 32583 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 18:08:35.455051 32583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "d36fde91c2d4471f8642db21358dccff"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "24e32b365902d3040cadae28257ebc1e"
server_key_iv: "82ff9a81d69b303d5c1266284ee0094d"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.455489 32583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:35.471757 32583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:35.476852 32583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:35.477113 32583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:35.477852 32583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:35.478883 32583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:35.478942 32583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.479000 32583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:35.479030 32583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.492991 32583 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:38163
I20260923 18:08:35.493211 32711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:38163 every 8 connection(s)
I20260923 18:08:35.494127 32583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:35.496294 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32583
I20260923 18:08:35.496409 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:08:35.496734 18611 external_mini_cluster.cc:1428] Setting key 0ec9011c7328f92e268784020f549634
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:35.527685 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.497983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:59625 (local address 127.18.44.254:45539)
0923 18:08:35.498931 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.498938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.500535 (+ 1597us) server_negotiation.cc:407] Connection header received
0923 18:08:35.503225 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.503231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.503381 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.503543 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:35.505352 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.506382 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.509366 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.509766 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.509927 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.519599 (+ 9672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:35.519605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:35.519609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:35.519668 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:35.522039 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.523029 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.523036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.523038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.523241 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.524703 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.524708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.524710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.525050 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:35.525165 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.527165 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.527479 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653}
I20260923 18:08:35.529633 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.499382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:59625)
0923 18:08:35.500287 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.500383 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.502660 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.503826 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.503837 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.504300 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:35.504942 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.504961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.506689 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.506695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.509032 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.509060 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.510070 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.510076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.510371 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.511606 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:35.511636 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:35.519051 (+ 7415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:35.522285 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.522294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.522320 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.522777 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.523884 (+ 1107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.523888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.523890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.524518 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.525928 (+ 1410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:35.525938 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:35.526654 (+ 716us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.526964 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.528091 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":288,"threads_started":1}
I20260923 18:08:35.531821 32712 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:35.532131 32712 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:35.533977 32712 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:35.536466 32383 ts_manager.cc:194] Registered new tserver with Master: d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163)
I20260923 18:08:35.537361 32383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:59625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:08:35.544538 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:35.687861 32717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:35.688277 32717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:35.688329 32717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:35.693953 32717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:35.694085 32717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:35.694154 32717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:35.701429 32717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:35.702699 32717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:35.703935 32717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:35.720745 32722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.720745 32725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:35.720741 32723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:35.722872 32717 server_base.cc:1034] running on GCE node
I20260923 18:08:35.723768 32717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:35.724920 32717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:35.726202 32717 hybrid_clock.cc:648] HybridClock initialized: now 1790186915726163 us; error 55 us; skew 500 ppm
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:35.732031 32717 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:35.733707 32717 webserver.cc:460] Webserver started at http://127.18.44.195:41157/ using document root <none> and password file <none>
I20260923 18:08:35.735631 32717 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:35.735957 32717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:35.736270 32717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:35.742309 32717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "5bf838db3331bda4de96609b77540d33"
server_key_iv: "fded2e758afcd5a3a18f376af8c1ad30"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.743083 32717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "5bf838db3331bda4de96609b77540d33"
server_key_iv: "fded2e758afcd5a3a18f376af8c1ad30"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.752471 32717 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260923 18:08:35.758085 32732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.760994 32717 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:08:35.761251 32717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "5bf838db3331bda4de96609b77540d33"
server_key_iv: "fded2e758afcd5a3a18f376af8c1ad30"
server_key_version: "encryptionkey@0"
I20260923 18:08:35.761474 32717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:35.801987 32717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:35.808458 32717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:35.808733 32717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:35.809607 32717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:35.810860 32717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:35.810925 32717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.810977 32717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:35.811009 32717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:35.825528 32717 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:43673
I20260923 18:08:35.825557 377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:43673 every 8 connection(s)
I20260923 18:08:35.826937 32717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:35.827561 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 32717
I20260923 18:08:35.827711 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:08:35.828020 18611 external_mini_cluster.cc:1428] Setting key 71d212f1191b978ef4bc4ab15d7e2719
Sep 23 18:08:35 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:35.845109 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.829853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:58083 (local address 127.18.44.254:45539)
0923 18:08:35.829992 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.829996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.831786 (+ 1790us) server_negotiation.cc:407] Connection header received
0923 18:08:35.833496 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.833500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.833580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.833657 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:35.835166 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.835783 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.837045 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.837275 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.837377 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.840594 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:35.840597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:35.840599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:35.840630 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:35.842535 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.843154 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.843157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.843159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.843245 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.843690 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.843692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.843694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.843903 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:35.843996 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.844771 (+ 775us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.844933 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:08:35.845978 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.830683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:58083)
0923 18:08:35.831556 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.831661 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.833229 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.833891 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.833900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.834343 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:35.834931 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.834947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.835926 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.835929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.836912 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.836923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.837451 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.837455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.837637 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.838347 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:35.838366 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:35.840366 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:35.842708 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.842715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.842732 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.843033 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.843355 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:35.843358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:35.843360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:35.843518 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:35.844019 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:35.844028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:35.844412 (+ 384us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.844633 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.845030 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":198,"threads_started":1}
I20260923 18:08:35.847971 378 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:35.848273 378 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:35.848875 378 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:35.850646 32383 ts_manager.cc:194] Registered new tserver with Master: 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
I20260923 18:08:35.851496 32383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:58083
I20260923 18:08:35.857805 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:08:35.871505 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.861314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.18.44.254:45539)
0923 18:08:35.861507 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.861512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.861711 (+ 199us) server_negotiation.cc:407] Connection header received
0923 18:08:35.861855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.861917 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.861970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.862047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:35.862825 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.863460 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.864328 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.864553 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.864648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.866256 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:35.866260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:35.866262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:35.866294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:35.868434 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.869105 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.869109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.869110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.869216 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:35.869752 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:35.869756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:35.869757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:35.870095 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:35.870280 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.871076 (+ 796us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.871245 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:08:35.876430 32383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:08:35.882155 32383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:08:35.902835 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.895023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.18.44.193:41739)
0923 18:08:35.896674 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.896683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.896714 (+ 31us) server_negotiation.cc:407] Connection header received
0923 18:08:35.896819 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.896826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.897098 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.897706 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:35.897854 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.899196 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.900832 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.901809 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.901972 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.902165 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.902551 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.902691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":167,"threads_started":1}
I20260923 18:08:35.902825 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.895016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:43673 (local address 127.0.0.1:51220)
0923 18:08:35.895837 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.895901 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.896058 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.897215 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.897221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.897249 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:35.897491 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.897502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.898930 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.898935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.900482 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.900508 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.902072 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.902076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.902297 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.902337 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.902424 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.902631 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":89,"spinlock_wait_cycles":19200,"thread_start_us":161,"threads_started":1}
I20260923 18:08:35.903620 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.894777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41739 (local address 127.0.0.1:38166)
0923 18:08:35.895837 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.895901 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.896058 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.897307 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.897311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.897333 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:35.897560 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.897568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.899408 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.899411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.900663 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.900671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.902022 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.902026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.902317 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.902358 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.902492 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.903501 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":1051,"spinlock_wait_cycles":3200,"thread_start_us":142,"threads_started":1}
I20260923 18:08:35.903628 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.895900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.18.44.195:43673)
0923 18:08:35.896448 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.896461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.896490 (+ 29us) server_negotiation.cc:407] Connection header received
0923 18:08:35.896701 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.896707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.896961 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.897659 (+ 698us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:35.897821 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.898717 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.900701 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.901809 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.902664 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.903239 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.903350 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.903470 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":225,"threads_started":1}
I20260923 18:08:35.905084 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.895805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.18.44.194:38163)
0923 18:08:35.896447 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.896461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.896995 (+ 534us) server_negotiation.cc:407] Connection header received
0923 18:08:35.897095 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.897101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.897318 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.897480 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:35.898017 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.899220 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.901391 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.902542 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.902661 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.903071 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.904430 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.904824 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":135,"threads_started":1}
I20260923 18:08:35.905532 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.895151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.0.0.1:58184)
0923 18:08:35.896735 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.896752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.896875 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.897716 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.897719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.897739 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:35.897841 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.897848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.899461 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.899465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.901211 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.901226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.903201 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.903205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.903393 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.903421 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.903527 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.903612 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"mutex_wait_us":249,"thread_start_us":150,"threads_started":1}
I20260923 18:08:35.907707 32512 tablet_service.cc:1467] Processing CreateTablet for tablet 952f92d7da6c4d969bf094453cf030f3 (DEFAULT_TABLE table=test-table [id=8caf61e9328941a6a97a71d2d5f4f535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:35.909534 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f92d7da6c4d969bf094453cf030f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:35.909754 312 tablet_service.cc:1467] Processing CreateTablet for tablet 952f92d7da6c4d969bf094453cf030f3 (DEFAULT_TABLE table=test-table [id=8caf61e9328941a6a97a71d2d5f4f535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:35.910192 32645 tablet_service.cc:1467] Processing CreateTablet for tablet 952f92d7da6c4d969bf094453cf030f3 (DEFAULT_TABLE table=test-table [id=8caf61e9328941a6a97a71d2d5f4f535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:35.911623 312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f92d7da6c4d969bf094453cf030f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:35.911619 32645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f92d7da6c4d969bf094453cf030f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:35.919418 396 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Bootstrap starting.
I20260923 18:08:35.923532 396 tablet_bootstrap.cc:654] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:35.924412 396 log.cc:824] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:35.926892 396 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: No bootstrap required, opened a new log
I20260923 18:08:35.927312 396 ts_tablet_manager.cc:1397] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 18:08:35.930150 399 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Bootstrap starting.
I20260923 18:08:35.930150 398 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Bootstrap starting.
I20260923 18:08:35.932183 396 raft_consensus.cc:348] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.932581 396 raft_consensus.cc:374] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:35.932641 396 raft_consensus.cc:729] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2b445b086e4ae79d0f7980251595db, State: Initialized, Role: FOLLOWER
I20260923 18:08:35.933182 396 consensus_queue.cc:260] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.933979 32578 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
I20260923 18:08:35.934471 396 ts_tablet_manager.cc:1428] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 18:08:35.935000 398 tablet_bootstrap.cc:654] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:35.935000 399 tablet_bootstrap.cc:654] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:35.937186 398 log.cc:824] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:35.937186 399 log.cc:824] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:35.940434 398 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: No bootstrap required, opened a new log
I20260923 18:08:35.940712 398 ts_tablet_manager.cc:1397] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 18:08:35.941525 399 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: No bootstrap required, opened a new log
I20260923 18:08:35.942052 399 ts_tablet_manager.cc:1397] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 18:08:35.944737 398 raft_consensus.cc:348] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.945052 398 raft_consensus.cc:374] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:35.945103 398 raft_consensus.cc:729] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36fde91c2d4471f8642db21358dccff, State: Initialized, Role: FOLLOWER
I20260923 18:08:35.945566 398 consensus_queue.cc:260] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.946427 32712 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
I20260923 18:08:35.946869 398 ts_tablet_manager.cc:1428] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:08:35.948432 399 raft_consensus.cc:348] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.948908 399 raft_consensus.cc:374] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:35.948980 399 raft_consensus.cc:729] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, State: Initialized, Role: FOLLOWER
I20260923 18:08:35.949791 399 consensus_queue.cc:260] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.950930 378 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
I20260923 18:08:35.951785 399 ts_tablet_manager.cc:1428] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260923 18:08:35.974642 400 raft_consensus.cc:482] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:35.974937 400 raft_consensus.cc:504] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.976198 400 leader_election.cc:288] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673), d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163)
I20260923 18:08:35.982758 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.976603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:34315 (local address 127.18.44.195:43673)
0923 18:08:35.976740 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.976745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.977123 (+ 378us) server_negotiation.cc:407] Connection header received
0923 18:08:35.977260 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.977264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.977465 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.977815 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:35.978236 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.979390 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.981164 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.981637 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.981766 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.981815 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.982493 (+ 678us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.982601 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:08:35.982734 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.976490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:43673 (local address 127.18.44.193:34315)
0923 18:08:35.976873 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.976889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.977070 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.977742 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.977746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.977797 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:35.978016 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.978027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.979620 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.979625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.980838 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.980861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.982113 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.982128 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.982268 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.982307 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.982404 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.982508 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":87,"spinlock_wait_cycles":18944,"thread_start_us":115,"threads_started":1}
I20260923 18:08:35.982944 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.976656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:56867 (local address 127.18.44.194:38163)
0923 18:08:35.976791 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:35.976796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:35.977175 (+ 379us) server_negotiation.cc:407] Connection header received
0923 18:08:35.977260 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:35.977264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:35.977465 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:35.977827 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:35.978236 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.979333 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.981197 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.981942 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.982233 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.982316 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:35.982692 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 18:08:35.982813 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:08:35.983426 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:35.976552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.18.44.193:56867)
0923 18:08:35.976919 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:35.976932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:35.977070 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:35.977742 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:35.977746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:35.977798 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:35.978048 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.978055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.979564 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.979569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.980842 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:35.980862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:35.982352 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:35.982356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:35.982509 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:35.982547 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 18:08:35.982654 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:08:35.982750 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":107,"spinlock_wait_cycles":36480,"thread_start_us":126,"threads_started":1}
I20260923 18:08:35.983929 332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f92d7da6c4d969bf094453cf030f3" candidate_uuid: "4e2b445b086e4ae79d0f7980251595db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" is_pre_election: true
I20260923 18:08:35.984364 32666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f92d7da6c4d969bf094453cf030f3" candidate_uuid: "4e2b445b086e4ae79d0f7980251595db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36fde91c2d4471f8642db21358dccff" is_pre_election: true
I20260923 18:08:35.984480 332 raft_consensus.cc:2454] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e2b445b086e4ae79d0f7980251595db in term 0.
I20260923 18:08:35.984690 32666 raft_consensus.cc:2454] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e2b445b086e4ae79d0f7980251595db in term 0.
I20260923 18:08:35.985162 32468 leader_election.cc:302] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, 4e2b445b086e4ae79d0f7980251595db; no voters:
I20260923 18:08:35.985637 400 raft_consensus.cc:2781] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:08:35.985778 400 raft_consensus.cc:482] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:08:35.985836 400 raft_consensus.cc:3037] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:35.987237 400 raft_consensus.cc:504] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:35.987839 400 leader_election.cc:288] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 1 election: Requested vote from peers 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673), d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163)
I20260923 18:08:35.988619 332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f92d7da6c4d969bf094453cf030f3" candidate_uuid: "4e2b445b086e4ae79d0f7980251595db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
I20260923 18:08:35.988822 332 raft_consensus.cc:3037] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:35.988801 32666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f92d7da6c4d969bf094453cf030f3" candidate_uuid: "4e2b445b086e4ae79d0f7980251595db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d36fde91c2d4471f8642db21358dccff"
I20260923 18:08:35.989120 32666 raft_consensus.cc:3037] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:35.990314 332 raft_consensus.cc:2454] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e2b445b086e4ae79d0f7980251595db in term 1.
I20260923 18:08:35.990712 32666 raft_consensus.cc:2454] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e2b445b086e4ae79d0f7980251595db in term 1.
I20260923 18:08:35.991014 32468 leader_election.cc:302] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, 4e2b445b086e4ae79d0f7980251595db; no voters:
I20260923 18:08:35.991430 400 raft_consensus.cc:2781] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:35.991854 400 raft_consensus.cc:686] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 LEADER]: Becoming Leader. State: Replica: 4e2b445b086e4ae79d0f7980251595db, State: Running, Role: LEADER
I20260923 18:08:35.992676 400 consensus_queue.cc:237] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
W20260923 18:08:35.996012 32713 tablet.cc:2369] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:08:35.997782 32387 catalog_manager.cc:5495] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e2b445b086e4ae79d0f7980251595db (127.18.44.193). New cstate: current_term: 1 leader_uuid: "4e2b445b086e4ae79d0f7980251595db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } health_report { overall_health: UNKNOWN } } }
I20260923 18:08:36.037815 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.032633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.18.44.193:41739)
0923 18:08:36.032918 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:36.032922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:36.032941 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:08:36.033038 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:36.033040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:36.033115 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:36.033241 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:08:36.033746 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.034429 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.035826 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.036062 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.036169 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.036789 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:08:36.037261 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:08:36.037375 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:36.037602 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 18:08:36.037684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260923 18:08:36.046664 332 raft_consensus.cc:1261] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Refusing update from remote peer 4e2b445b086e4ae79d0f7980251595db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:36.046993 32666 raft_consensus.cc:1261] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Refusing update from remote peer 4e2b445b086e4ae79d0f7980251595db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:36.047881 400 consensus_queue.cc:1035] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:36.048308 407 consensus_queue.cc:1035] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:36.060698 411 mvcc.cc:205] Tried to move back new op lower bound from 7332605608116260864 to 7332605607911309312. Current Snapshot: MvccSnapshot[applied={T|T < 7332605608116260864}]
I20260923 18:08:36.073271 32386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:36.073551 32386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:36.077591 32386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Sending DeleteTablet for 3 replicas of tablet 952f92d7da6c4d969bf094453cf030f3
I20260923 18:08:36.078871 32512 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f92d7da6c4d969bf094453cf030f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260923 18:08:36.079608 420 tablet_replica.cc:330] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: stopping tablet replica
I20260923 18:08:36.080045 420 raft_consensus.cc:2229] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:08:36.079970 312 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f92d7da6c4d969bf094453cf030f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:51220
I20260923 18:08:36.079970 32645 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f92d7da6c4d969bf094453cf030f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:58184
I20260923 18:08:36.080595 420 raft_consensus.cc:2258] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:36.080767 422 tablet_replica.cc:330] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: stopping tablet replica
I20260923 18:08:36.081101 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32449
I20260923 18:08:36.083146 422 raft_consensus.cc:2229] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:36.083612 422 raft_consensus.cc:2258] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:36.084764 421 tablet_replica.cc:330] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: stopping tablet replica
I20260923 18:08:36.085361 421 raft_consensus.cc:2229] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:36.086032 421 raft_consensus.cc:2258] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:36.086472 422 ts_tablet_manager.cc:1905] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:08:36.088215 421 ts_tablet_manager.cc:1905] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:08:36.089756 32373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:41739 (error 108)
I20260923 18:08:36.089851 422 ts_tablet_manager.cc:1918] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:36.089967 422 log.cc:1196] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/952f92d7da6c4d969bf094453cf030f3
I20260923 18:08:36.090283 422 ts_tablet_manager.cc:1939] T 952f92d7da6c4d969bf094453cf030f3 P d36fde91c2d4471f8642db21358dccff: Deleting consensus metadata
I20260923 18:08:36.090914 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32583
I20260923 18:08:36.092382 421 ts_tablet_manager.cc:1918] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:36.092578 421 log.cc:1196] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/952f92d7da6c4d969bf094453cf030f3
I20260923 18:08:36.093029 421 ts_tablet_manager.cc:1939] T 952f92d7da6c4d969bf094453cf030f3 P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting consensus metadata
I20260923 18:08:36.094276 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.093936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41739 (local address 127.0.0.1:38170)
0923 18:08:36.094099 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:36.094180 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 18:08:36.094558 32373 catalog_manager.cc:4584] TS 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 952f92d7da6c4d969bf094453cf030f3: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41739: connect: Connection refused (error 111)
I20260923 18:08:36.095036 32374 catalog_manager.cc:4841] TS 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): tablet 952f92d7da6c4d969bf094453cf030f3 (table test-table [id=8caf61e9328941a6a97a71d2d5f4f535]) successfully deleted
I20260923 18:08:36.098893 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32717
I20260923 18:08:36.101152 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.099362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.0.0.1:58194)
0923 18:08:36.100858 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:36.101030 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1383}
I20260923 18:08:36.106168 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 32355
I20260923 18:08:36.113384 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--webserver_interface=127.18.44.254
--webserver_port=39025
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:36.236933 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:36.237419 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:36.237538 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:36.242170 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:36.242277 424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:36.242297 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:36.242313 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:36.242328 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:36.249372 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:36.250631 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:36.251809 424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:36.268460 430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:36.268496 433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:36.268702 424 server_base.cc:1034] running on GCE node
W20260923 18:08:36.268465 431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:36.269779 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:36.271335 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:36.274865 424 hybrid_clock.cc:648] HybridClock initialized: now 1790186916274757 us; error 122 us; skew 500 ppm
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:36.278741 424 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:36.279997 424 webserver.cc:460] Webserver started at http://127.18.44.254:39025/ using document root <none> and password file <none>
I20260923 18:08:36.280719 424 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:36.280827 424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:36.286104 424 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:08:36.289283 440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:36.291152 424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260923 18:08:36.291379 424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "080cf4f857144262a8f16c4677a1ba20"
format_stamp: "Formatted at 2026-09-23 18:08:34 on dist-test-slave-rvjj"
server_key: "abc6f1533d44c3232db6ca8620da6086"
server_key_iv: "66033d93a83504820604311d98b82ae1"
server_key_version: "encryptionkey@0"
I20260923 18:08:36.292086 424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:36.311164 424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:36.314556 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:36.314754 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:36.323477 424 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45539
I20260923 18:08:36.323859 492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45539 every 8 connection(s)
I20260923 18:08:36.325045 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:36.332476 493 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:08:36.333343 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 424
I20260923 18:08:36.333942 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:41739
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=46263
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:36.336237 493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap starting.
I20260923 18:08:36.363585 493 log.cc:824] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:36.377389 493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:08:36.378028 493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap complete.
I20260923 18:08:36.385957 493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:36.386677 493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Initialized, Role: FOLLOWER
I20260923 18:08:36.387487 493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:36.387691 493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:36.387792 493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:36.387913 493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:08:36.391589 493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:36.392192 493 leader_election.cc:302] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080cf4f857144262a8f16c4677a1ba20; no voters:
I20260923 18:08:36.392699 493 leader_election.cc:288] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:08:36.392901 497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:08:36.393302 497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 LEADER]: Becoming Leader. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Running, Role: LEADER
I20260923 18:08:36.393923 497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:36.394598 493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:36.396673 499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080cf4f857144262a8f16c4677a1ba20. Latest consensus state: current_term: 2 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:36.397099 499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:36.397442 498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:36.397658 498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:36.398747 506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:36.405516 506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:36.406383 506 catalog_manager.cc:1259] Loaded cluster ID: 710008fc681c448f8ab61b528aa2dcb8
I20260923 18:08:36.406486 506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:36.408515 506 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 18:08:36.413096 506 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 18:08:36.420174 506 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:36.421257 506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Loaded TSK: 0
I20260923 18:08:36.422782 506 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:08:36.481649 495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:36.482167 495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:36.482293 495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:36.487010 495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:36.487121 495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:36.487186 495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:36.493407 495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:41739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=46263
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:36.494679 495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:36.496210 495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:36.510972 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:36.510923 521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:36.510910 523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:36.511745 495 server_base.cc:1034] running on GCE node
I20260923 18:08:36.512302 495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:36.513213 495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:36.516636 495 hybrid_clock.cc:648] HybridClock initialized: now 1790186916516585 us; error 55 us; skew 500 ppm
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:36.520294 495 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:36.521564 495 webserver.cc:460] Webserver started at http://127.18.44.193:46263/ using document root <none> and password file <none>
I20260923 18:08:36.522228 495 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:36.522337 495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:36.529099 495 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:08:36.532384 530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:36.533927 495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:36.534097 495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "4e2b445b086e4ae79d0f7980251595db"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "c3a32a6368277024106b3905f2ac20a2"
server_key_iv: "6bbd4a1ce44cb4583ff70bb8450cfd93"
server_key_version: "encryptionkey@0"
I20260923 18:08:36.535112 495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:36.559386 495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:36.563290 495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:36.563511 495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:36.564255 495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:36.565821 537 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:08:36.571184 495 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:08:36.571509 495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260923 18:08:36.571616 495 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:08:36.574047 495 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:08:36.574280 495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:36.574502 537 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Bootstrap starting.
I20260923 18:08:36.590787 495 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:41739
I20260923 18:08:36.591004 644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:41739 every 8 connection(s)
I20260923 18:08:36.592617 495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:36.596486 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 495
I20260923 18:08:36.597174 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:38163
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=43105
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:36.612974 537 log.cc:824] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:36.625244 537 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:08:36.625890 537 tablet_bootstrap.cc:492] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Bootstrap complete.
I20260923 18:08:36.627171 537 ts_tablet_manager.cc:1397] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.024s
I20260923 18:08:36.630414 537 raft_consensus.cc:348] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:36.631125 537 raft_consensus.cc:729] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2b445b086e4ae79d0f7980251595db, State: Initialized, Role: FOLLOWER
I20260923 18:08:36.631842 537 consensus_queue.cc:260] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } }
I20260923 18:08:36.632697 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.601877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:48037)
0923 18:08:36.607411 (+ 5534us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:36.607517 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:36.609184 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:36.613731 (+ 4547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:36.613744 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:36.614622 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:36.615451 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.615482 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.618097 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.618101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.620060 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.620078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.621379 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.621383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.621616 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.622379 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:36.622401 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:36.624623 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:36.628244 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.628251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.628268 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.628710 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.629442 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.629445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.629447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.629729 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.630660 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:36.630666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:36.631068 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 18:08:36.631314 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 18:08:36.631689 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5108,"thread_start_us":347,"threads_started":1}
I20260923 18:08:36.633100 537 ts_tablet_manager.cc:1428] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:08:36.633441 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.602398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:48037 (local address 127.18.44.254:45539)
0923 18:08:36.610291 (+ 7893us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:36.610316 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:36.610372 (+ 56us) server_negotiation.cc:407] Connection header received
0923 18:08:36.611228 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:36.611279 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:36.612109 (+ 830us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:36.612639 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:36.615759 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.617748 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.620278 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.621159 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.621395 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.625040 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:36.625095 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:36.625114 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:36.625159 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:36.628085 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.629087 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.629098 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.629104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.629311 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.629886 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.629890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.629892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.630453 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:36.630626 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:36.631497 (+ 871us) server_negotiation.cc:299] Negotiation successful
0923 18:08:36.631835 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7664,"thread_start_us":209,"threads_started":1}
I20260923 18:08:36.636726 645 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:36.637020 645 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:36.637626 645 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:36.641317 457 ts_manager.cc:194] Registered new tserver with Master: 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739)
I20260923 18:08:36.645432 457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:48037
I20260923 18:08:36.648782 645 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:08:36.663143 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.644849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41739 (local address 127.0.0.1:38178)
0923 18:08:36.645800 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:36.645849 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:36.646090 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:36.646692 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:36.646697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:36.647214 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:36.647395 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.647405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.648830 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.648834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.650367 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.650389 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.651856 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.651860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.651984 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.653135 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:36.653151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:36.656485 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:36.660179 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.660194 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.660200 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.660695 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.661431 (+ 736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.661435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.661436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.661551 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.662453 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:36.662459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:36.662609 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 18:08:36.662748 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 18:08:36.662958 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":159,"threads_started":1}
I20260923 18:08:36.663676 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.645107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.18.44.193:41739)
0923 18:08:36.645749 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:36.645769 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:36.646046 (+ 277us) server_negotiation.cc:407] Connection header received
0923 18:08:36.646284 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:36.646290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:36.646507 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:36.646671 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:36.647547 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.648653 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.650644 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.651207 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.651727 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.656709 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:36.656725 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:36.656740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:36.656775 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:36.659939 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.660863 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.660871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.660873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.661179 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.661656 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.661660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.661661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.662171 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:36.662446 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:36.662983 (+ 537us) server_negotiation.cc:299] Negotiation successful
0923 18:08:36.663218 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":121,"threads_started":1}
I20260923 18:08:36.665000 578 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f92d7da6c4d969bf094453cf030f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:36 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:38178
I20260923 18:08:36.666142 654 tablet_replica.cc:330] stopping tablet replica
I20260923 18:08:36.666510 654 raft_consensus.cc:2229] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:36.666810 654 raft_consensus.cc:2258] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:36.668041 654 ts_tablet_manager.cc:1905] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:08:36.675199 654 ts_tablet_manager.cc:1918] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:36.675423 654 log.cc:1196] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/952f92d7da6c4d969bf094453cf030f3
I20260923 18:08:36.675891 654 ts_tablet_manager.cc:1939] T 952f92d7da6c4d969bf094453cf030f3 P 4e2b445b086e4ae79d0f7980251595db: Deleting consensus metadata
I20260923 18:08:36.678170 443 catalog_manager.cc:4841] TS 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739): tablet 952f92d7da6c4d969bf094453cf030f3 (table test-table [id=8caf61e9328941a6a97a71d2d5f4f535]) successfully deleted
W20260923 18:08:36.779901 647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:36.780244 647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:36.780292 647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:36.785516 647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:36.785630 647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:36.785698 647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:36.792481 647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:38163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=43105
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:36.794377 647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:36.795928 647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:36.808719 659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:36.808791 647 server_base.cc:1034] running on GCE node
W20260923 18:08:36.808720 660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:36.808719 662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:36.810046 647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:36.810791 647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:36.812008 647 hybrid_clock.cc:648] HybridClock initialized: now 1790186916811967 us; error 58 us; skew 500 ppm
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:36.815693 647 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:36.817176 647 webserver.cc:460] Webserver started at http://127.18.44.194:43105/ using document root <none> and password file <none>
I20260923 18:08:36.817835 647 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:36.817938 647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:36.823014 647 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 18:08:36.826390 669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:36.828214 647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:36.828379 647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "d36fde91c2d4471f8642db21358dccff"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "24e32b365902d3040cadae28257ebc1e"
server_key_iv: "82ff9a81d69b303d5c1266284ee0094d"
server_key_version: "encryptionkey@0"
I20260923 18:08:36.829129 647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:36.857549 647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:36.862828 647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:36.863039 647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:36.864003 647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:36.865264 647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:36.865336 647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:36.865381 647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:36.865399 647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:36.878480 647 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:38163
I20260923 18:08:36.879434 782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:38163 every 8 connection(s)
I20260923 18:08:36.879825 647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:36.884071 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 647
I20260923 18:08:36.884615 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:43673
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=41157
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:08:36 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186916, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:36.900816 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.882087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:50485 (local address 127.18.44.254:45539)
0923 18:08:36.882375 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:36.882379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:36.883842 (+ 1463us) server_negotiation.cc:407] Connection header received
0923 18:08:36.885818 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:36.885821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:36.885887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:36.886051 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:36.887739 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.888528 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.890167 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.890571 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.890765 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.894702 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:36.894708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:36.894712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:36.894756 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:36.897313 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.898623 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.898628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.898630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.898760 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:36.899223 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:36.899227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:36.899229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:36.899525 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:36.899639 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:36.900443 (+ 804us) server_negotiation.cc:299] Negotiation successful
0923 18:08:36.900611 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:08:36.901646 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:36.882863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:50485)
0923 18:08:36.883622 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:36.883692 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:36.885390 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:36.886320 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:36.886333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:36.886795 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:36.887417 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.887436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.888735 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.888738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.889901 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:36.889916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:36.890872 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:36.890882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:36.891098 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:36.891976 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:36.891996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:36.894399 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:36.897574 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.897581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.897600 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.898229 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.898901 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:36.898904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:36.898905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:36.899101 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:36.899729 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:36.899735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:36.900083 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 18:08:36.900277 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 18:08:36.900603 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":171,"threads_started":1}
I20260923 18:08:36.902949 783 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:36.903221 783 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:36.903880 783 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:36.905416 457 ts_manager.cc:194] Registered new tserver with Master: d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163)
I20260923 18:08:36.906085 457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:50485
W20260923 18:08:37.029528 786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:37.029877 786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:37.029927 786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:37.034765 786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:37.034885 786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:37.034973 786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:37.042635 786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=41157
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:37.043969 786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:37.045077 786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:37.056788 792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:37.056716 791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:37.056819 794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:37.056711 786 server_base.cc:1034] running on GCE node
I20260923 18:08:37.057513 786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:37.058588 786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:37.059795 786 hybrid_clock.cc:648] HybridClock initialized: now 1790186917059714 us; error 101 us; skew 500 ppm
Sep 23 18:08:37 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:37.063505 786 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:37.064777 786 webserver.cc:460] Webserver started at http://127.18.44.195:41157/ using document root <none> and password file <none>
I20260923 18:08:37.065480 786 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:37.065589 786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:37.071066 786 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:08:37.073899 801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:37.075736 786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:37.075907 786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "5bf838db3331bda4de96609b77540d33"
server_key_iv: "fded2e758afcd5a3a18f376af8c1ad30"
server_key_version: "encryptionkey@0"
I20260923 18:08:37.076537 786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:37.096473 786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:37.100240 786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:37.100457 786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:37.101106 786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:37.102015 786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:37.102066 786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:37.102136 786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:37.102162 786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:37.111994 786 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:43673
I20260923 18:08:37.112079 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:43673 every 8 connection(s)
I20260923 18:08:37.113179 786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:37.114928 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 786
Sep 23 18:08:37 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:37.125891 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.116167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.18.44.254:45539)
0923 18:08:37.116385 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.116389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.116404 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:08:37.116452 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.116454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.116503 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.116598 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:37.117667 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.118413 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.119754 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.120003 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.120099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.122154 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:37.122158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:37.122160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:37.122196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:37.124030 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:37.124485 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:37.124488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:37.124489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:37.124583 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:37.124840 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:37.124843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:37.124845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:37.125050 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:37.125251 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.125484 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.125707 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260923 18:08:37.130121 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.115429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:48005 (local address 127.18.44.254:45539)
0923 18:08:37.115605 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.115610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.116769 (+ 1159us) server_negotiation.cc:407] Connection header received
0923 18:08:37.118550 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.118553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.118613 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.118679 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:37.120127 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.120623 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.121874 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.122236 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.122357 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.125227 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:37.125230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:37.125232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:37.125255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:37.126966 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:37.128086 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:37.128089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:37.128091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:37.128211 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:37.128756 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:37.128759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:37.128760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:37.129008 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:37.129107 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.129778 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.129993 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:08:37.131585 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.115894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:48005)
0923 18:08:37.116593 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.116658 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.118250 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.118913 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:37.118921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:37.119396 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:37.119972 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.119989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.120756 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.120759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.121717 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.121730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.122444 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.122449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.122701 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.123309 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:37.123333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:37.125036 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:37.127218 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:37.127225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:37.127244 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:37.127887 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:37.128316 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:37.128319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:37.128322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:37.128618 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:37.129161 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:37.129167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:37.129455 (+ 288us) client_negotiation.cc:769] Sending connection context
0923 18:08:37.129640 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 18:08:37.129945 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":157,"threads_started":1}
I20260923 18:08:37.133205 915 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45539
I20260923 18:08:37.133504 915 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:37.134213 915 heartbeater.cc:511] Master 127.18.44.254:45539 requested a full tablet report, sending...
I20260923 18:08:37.135569 457 ts_manager.cc:194] Registered new tserver with Master: 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
I20260923 18:08:37.136183 457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:48005
I20260923 18:08:37.141499 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:08:37.146706 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.142264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.18.44.254:45539)
0923 18:08:37.142385 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.142389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.142580 (+ 191us) server_negotiation.cc:407] Connection header received
0923 18:08:37.142679 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.142681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.142797 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.142900 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:08:37.143307 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.143995 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.145018 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.145248 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.145340 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.145719 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:08:37.146192 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:08:37.146323 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.146452 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.146533 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:08:37.147632 457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:08:37.152688 457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:08:37.162019 578 tablet_service.cc:1467] Processing CreateTablet for tablet 6c185515656849a380217430c2d5f0ff (DEFAULT_TABLE table=test-table [id=12e74d8b83ee4ac2ab4da9121186bc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:37.162730 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c185515656849a380217430c2d5f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:37.166465 927 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Bootstrap starting.
I20260923 18:08:37.166855 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.161085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.0.0.1:58202)
0923 18:08:37.161287 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.161304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.161398 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.162369 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:37.162373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:37.162392 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:37.162507 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.162514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.163825 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.163829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.165048 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.165061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.166378 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.166382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.166509 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.166553 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 18:08:37.166624 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 18:08:37.166718 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 18:08:37.166942 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.161260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.18.44.194:38163)
0923 18:08:37.161749 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.161765 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.161792 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:08:37.161892 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.161898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.162151 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.162373 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:37.162672 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.163692 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.165271 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.166248 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.166456 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.166640 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.166697 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.166795 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
I20260923 18:08:37.167774 927 tablet_bootstrap.cc:654] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:37.168982 927 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: No bootstrap required, opened a new log
I20260923 18:08:37.169085 927 ts_tablet_manager.cc:1397] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:08:37.169504 927 raft_consensus.cc:348] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.169621 927 raft_consensus.cc:374] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:37.169656 927 raft_consensus.cc:729] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2b445b086e4ae79d0f7980251595db, State: Initialized, Role: FOLLOWER
I20260923 18:08:37.169836 927 consensus_queue.cc:260] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.170233 927 ts_tablet_manager.cc:1428] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:08:37.170290 717 tablet_service.cc:1467] Processing CreateTablet for tablet 6c185515656849a380217430c2d5f0ff (DEFAULT_TABLE table=test-table [id=12e74d8b83ee4ac2ab4da9121186bc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:37.171483 717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c185515656849a380217430c2d5f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:37.171937 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.161240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:43673 (local address 127.0.0.1:51228)
0923 18:08:37.166269 (+ 5029us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.166284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.166491 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.167360 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:37.167363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:37.167383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:37.167493 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.167499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.168930 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.168934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.170037 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.170051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.171475 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.171479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.171578 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.171607 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 18:08:37.171688 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:08:37.171791 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4946,"thread_start_us":127,"threads_started":1}
I20260923 18:08:37.171998 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.161842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.18.44.195:43673)
0923 18:08:37.162233 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.162245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.166392 (+ 4147us) server_negotiation.cc:407] Connection header received
0923 18:08:37.166668 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.166675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.166943 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.167103 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:37.167629 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.168752 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.170486 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.171322 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.171448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.171604 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.171736 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.171846 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":172,"threads_started":1}
I20260923 18:08:37.175305 849 tablet_service.cc:1467] Processing CreateTablet for tablet 6c185515656849a380217430c2d5f0ff (DEFAULT_TABLE table=test-table [id=12e74d8b83ee4ac2ab4da9121186bc9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:37.176561 849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c185515656849a380217430c2d5f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:37.180747 930 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Bootstrap starting.
I20260923 18:08:37.183730 930 tablet_bootstrap.cc:654] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:37.184701 930 log.cc:824] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:37.185830 931 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Bootstrap starting.
I20260923 18:08:37.187088 930 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: No bootstrap required, opened a new log
I20260923 18:08:37.187347 930 ts_tablet_manager.cc:1397] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260923 18:08:37.188515 931 tablet_bootstrap.cc:654] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:37.190024 931 log.cc:824] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:37.191926 930 raft_consensus.cc:348] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.192287 930 raft_consensus.cc:374] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:37.192361 930 raft_consensus.cc:729] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36fde91c2d4471f8642db21358dccff, State: Initialized, Role: FOLLOWER
I20260923 18:08:37.193014 931 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: No bootstrap required, opened a new log
I20260923 18:08:37.192929 930 consensus_queue.cc:260] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.193250 931 ts_tablet_manager.cc:1397] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 18:08:37.193831 783 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
I20260923 18:08:37.194180 930 ts_tablet_manager.cc:1428] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:08:37.197297 931 raft_consensus.cc:348] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.197664 931 raft_consensus.cc:374] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:37.197739 931 raft_consensus.cc:729] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, State: Initialized, Role: FOLLOWER
I20260923 18:08:37.198351 931 consensus_queue.cc:260] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.199236 915 heartbeater.cc:503] Master 127.18.44.254:45539 was elected leader, sending a full tablet report...
I20260923 18:08:37.199815 931 ts_tablet_manager.cc:1428] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260923 18:08:37.263639 934 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:37.263897 934 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.265213 934 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673), 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739)
I20260923 18:08:37.269033 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.265421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:43673 (local address 127.18.44.194:40989)
0923 18:08:37.265650 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.265665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.265771 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.266112 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:37.266115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:37.266163 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:37.266265 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.266271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.267013 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.267016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.267853 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.267863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.268600 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.268604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.268714 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.268733 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:08:37.268799 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 18:08:37.268882 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"spinlock_wait_cycles":3200}
I20260923 18:08:37.269033 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.265560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:40989 (local address 127.18.44.195:43673)
0923 18:08:37.265779 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.265785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.265806 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:08:37.265942 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.265945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.265994 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.266072 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:37.266401 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.266904 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.267990 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.268481 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.268600 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.268656 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.268833 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.268919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260923 18:08:37.269761 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c185515656849a380217430c2d5f0ff" candidate_uuid: "d36fde91c2d4471f8642db21358dccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" is_pre_election: true
I20260923 18:08:37.269913 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.265736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:59937 (local address 127.18.44.193:41739)
0923 18:08:37.266006 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.266011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.266127 (+ 116us) server_negotiation.cc:407] Connection header received
0923 18:08:37.266272 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.266275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.266331 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.266426 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:37.266797 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.267397 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.268651 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.269284 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.269398 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.269580 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.269709 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.269784 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
I20260923 18:08:37.269912 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.265552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41739 (local address 127.18.44.194:59937)
0923 18:08:37.266027 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.266040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.266166 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.266498 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:37.266516 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:37.266539 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:37.266667 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.266673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.267574 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.267577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.268509 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:37.268521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.269469 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.269472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:37.269571 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.269594 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:08:37.269666 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:08:37.269743 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":60,"spinlock_wait_cycles":30336,"thread_start_us":136,"threads_started":1}
I20260923 18:08:37.270115 869 raft_consensus.cc:2454] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36fde91c2d4471f8642db21358dccff in term 0.
I20260923 18:08:37.270458 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c185515656849a380217430c2d5f0ff" candidate_uuid: "d36fde91c2d4471f8642db21358dccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2b445b086e4ae79d0f7980251595db" is_pre_election: true
I20260923 18:08:37.270735 599 raft_consensus.cc:2454] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d36fde91c2d4471f8642db21358dccff in term 0.
I20260923 18:08:37.270779 673 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff; no voters:
I20260923 18:08:37.271116 934 raft_consensus.cc:2781] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:08:37.271199 934 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:08:37.271245 934 raft_consensus.cc:3037] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:37.272370 934 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.272771 934 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [CANDIDATE]: Term 1 election: Requested vote from peers 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673), 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739)
I20260923 18:08:37.273324 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c185515656849a380217430c2d5f0ff" candidate_uuid: "d36fde91c2d4471f8642db21358dccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2b445b086e4ae79d0f7980251595db"
I20260923 18:08:37.273370 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c185515656849a380217430c2d5f0ff" candidate_uuid: "d36fde91c2d4471f8642db21358dccff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
I20260923 18:08:37.273460 599 raft_consensus.cc:3037] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:37.273572 869 raft_consensus.cc:3037] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:37.274574 599 raft_consensus.cc:2454] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36fde91c2d4471f8642db21358dccff in term 1.
I20260923 18:08:37.274876 869 raft_consensus.cc:2454] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d36fde91c2d4471f8642db21358dccff in term 1.
I20260923 18:08:37.275009 672 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db, d36fde91c2d4471f8642db21358dccff; no voters:
I20260923 18:08:37.275218 934 raft_consensus.cc:2781] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:37.275506 934 raft_consensus.cc:686] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 1 LEADER]: Becoming Leader. State: Replica: d36fde91c2d4471f8642db21358dccff, State: Running, Role: LEADER
I20260923 18:08:37.275921 934 consensus_queue.cc:237] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.280115 456 catalog_manager.cc:5495] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff reported cstate change: term changed from 0 to 1, leader changed from <none> to d36fde91c2d4471f8642db21358dccff (127.18.44.194). New cstate: current_term: 1 leader_uuid: "d36fde91c2d4471f8642db21358dccff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } health_report { overall_health: UNKNOWN } } }
I20260923 18:08:37.305040 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.300700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.18.44.194:38163)
0923 18:08:37.300970 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:37.300976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:37.301050 (+ 74us) server_negotiation.cc:407] Connection header received
0923 18:08:37.301240 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:37.301243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:37.301470 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:37.301687 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:08:37.302093 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.302611 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.303501 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:37.303718 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:37.303813 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:37.304141 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:08:37.304604 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:08:37.304719 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:37.304803 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 18:08:37.304874 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260923 18:08:37.311880 869 raft_consensus.cc:1261] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Refusing update from remote peer d36fde91c2d4471f8642db21358dccff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:37.312039 599 raft_consensus.cc:1261] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Refusing update from remote peer d36fde91c2d4471f8642db21358dccff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:37.312762 938 consensus_queue.cc:1035] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:37.313117 934 consensus_queue.cc:1035] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:37.335070 457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:37.335305 457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:37.337742 457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Sending DeleteTablet for 3 replicas of tablet 6c185515656849a380217430c2d5f0ff
I20260923 18:08:37.338637 849 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c185515656849a380217430c2d5f0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:37 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:51228
I20260923 18:08:37.338768 578 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c185515656849a380217430c2d5f0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:37 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:38178
I20260923 18:08:37.339233 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 495
I20260923 18:08:37.339273 717 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c185515656849a380217430c2d5f0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:37 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:58202
I20260923 18:08:37.343638 952 tablet_replica.cc:330] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: stopping tablet replica
I20260923 18:08:37.343739 953 tablet_replica.cc:330] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: stopping tablet replica
I20260923 18:08:37.344156 953 raft_consensus.cc:2229] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:08:37.344164 952 raft_consensus.cc:2229] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:37.344811 953 raft_consensus.cc:2258] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:37.344846 952 raft_consensus.cc:2258] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:37.347033 953 ts_tablet_manager.cc:1905] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:08:37.347350 952 ts_tablet_manager.cc:1905] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:08:37.349901 443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:41739 (error 108)
I20260923 18:08:37.350752 952 ts_tablet_manager.cc:1918] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:37.350932 953 ts_tablet_manager.cc:1918] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:37.351071 953 log.cc:1196] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/6c185515656849a380217430c2d5f0ff
I20260923 18:08:37.351284 952 log.cc:1196] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/6c185515656849a380217430c2d5f0ff
I20260923 18:08:37.351449 953 ts_tablet_manager.cc:1939] T 6c185515656849a380217430c2d5f0ff P d36fde91c2d4471f8642db21358dccff: Deleting consensus metadata
I20260923 18:08:37.351830 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 647
I20260923 18:08:37.352149 952 ts_tablet_manager.cc:1939] T 6c185515656849a380217430c2d5f0ff P 3cf3bd0f3cdd4d9eafb26b507ee75bd7: Deleting consensus metadata
I20260923 18:08:37.353943 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.353635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41739 (local address 127.0.0.1:38186)
0923 18:08:37.353757 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.353834 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 18:08:37.354086 444 catalog_manager.cc:4841] TS 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): tablet 6c185515656849a380217430c2d5f0ff (table test-table [id=12e74d8b83ee4ac2ab4da9121186bc9e]) successfully deleted
W20260923 18:08:37.354554 443 catalog_manager.cc:4584] TS 4e2b445b086e4ae79d0f7980251595db (127.18.44.193:41739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c185515656849a380217430c2d5f0ff: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41739: connect: Connection refused (error 111)
I20260923 18:08:37.359058 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 786
I20260923 18:08:37.360527 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.359299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.0.0.1:58218)
0923 18:08:37.360344 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.360420 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":940}
I20260923 18:08:37.365240 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 424
I20260923 18:08:37.372049 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--webserver_interface=127.18.44.254
--webserver_port=39025
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:37.513200 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:37.513667 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:37.513814 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:37.519107 955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:37.519502 955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:37.519645 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:37.519667 955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:37.519683 955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:37.527462 955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:37.528645 955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:37.529971 955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:37.540756 963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:37.540796 961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:37.540815 955 server_base.cc:1034] running on GCE node
W20260923 18:08:37.541002 960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:37.541813 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:37.543154 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:37.544413 955 hybrid_clock.cc:648] HybridClock initialized: now 1790186917544388 us; error 43 us; skew 500 ppm
Sep 23 18:08:37 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:37.547958 955 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:37.549305 955 webserver.cc:460] Webserver started at http://127.18.44.254:39025/ using document root <none> and password file <none>
I20260923 18:08:37.550128 955 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:37.550217 955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:37.555653 955 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:37.558171 970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:37.559849 955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:37.560019 955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "080cf4f857144262a8f16c4677a1ba20"
format_stamp: "Formatted at 2026-09-23 18:08:34 on dist-test-slave-rvjj"
server_key: "abc6f1533d44c3232db6ca8620da6086"
server_key_iv: "66033d93a83504820604311d98b82ae1"
server_key_version: "encryptionkey@0"
I20260923 18:08:37.560645 955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:37.575644 955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:37.579996 955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:37.580232 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:37.587621 955 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45539
I20260923 18:08:37.587883 1022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45539 every 8 connection(s)
I20260923 18:08:37.588766 955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:37.591534 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 955
I20260923 18:08:37.592026 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:41739
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=46263
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:37.594267 1023 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:08:37.596585 1023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap starting.
I20260923 18:08:37.615725 1023 log.cc:824] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:37.631677 1023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:08:37.632185 1023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20: Bootstrap complete.
I20260923 18:08:37.636737 1023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:37.637259 1023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Initialized, Role: FOLLOWER
I20260923 18:08:37.637812 1023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:37.637960 1023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:37.638031 1023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:37.638129 1023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:08:37.640336 1023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:37.640700 1023 leader_election.cc:302] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080cf4f857144262a8f16c4677a1ba20; no voters:
I20260923 18:08:37.641042 1023 leader_election.cc:288] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 18:08:37.641194 1027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 18:08:37.641444 1027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [term 3 LEADER]: Becoming Leader. State: Replica: 080cf4f857144262a8f16c4677a1ba20, State: Running, Role: LEADER
I20260923 18:08:37.641779 1027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } }
I20260923 18:08:37.642395 1023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:37.643429 1028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:37.643567 1028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:37.643819 1029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080cf4f857144262a8f16c4677a1ba20. Latest consensus state: current_term: 3 leader_uuid: "080cf4f857144262a8f16c4677a1ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cf4f857144262a8f16c4677a1ba20" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45539 } } }
I20260923 18:08:37.643901 1029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cf4f857144262a8f16c4677a1ba20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:37.644522 1036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:37.649737 1036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:37.650439 1036 catalog_manager.cc:1259] Loaded cluster ID: 710008fc681c448f8ab61b528aa2dcb8
I20260923 18:08:37.650506 1036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 18:08:37.651433 1036 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc2d23c59ad google::LogMessage::Fail() at ??:0
@ 0x7fc2d23c888a google::LogMessage::SendToLog() at ??:0
@ 0x7fc2d23c54de google::LogMessage::Flush() at ??:0
@ 0x7fc2d23c6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc2dcd08fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc2dcd09d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc2dcd06579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc2dcd4a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc2d46173d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc2d4617c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc2d46193f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc2d4608512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc2d5c4c6db start_thread at ??:0
@ 0x7fc2d0ec361f clone at ??:0
W20260923 18:08:37.753540 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:37.753871 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:37.753944 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:37.759367 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:37.759487 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:37.759645 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:37.767940 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:41739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=46263
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:37.769124 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:37.770190 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:37.782809 1065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:37.782814 1064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:37.782924 1067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:37.784117 1025 server_base.cc:1034] running on GCE node
I20260923 18:08:37.784771 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:37.785530 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:37.788928 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790186917788933 us; error 107 us; skew 500 ppm
Sep 23 18:08:37 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:37.793262 1025 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:37.794543 1025 webserver.cc:460] Webserver started at http://127.18.44.193:46263/ using document root <none> and password file <none>
I20260923 18:08:37.795174 1025 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:37.795288 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:37.800431 1025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 18:08:37.803310 1074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:37.805333 1025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:08:37.805529 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "4e2b445b086e4ae79d0f7980251595db"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "c3a32a6368277024106b3905f2ac20a2"
server_key_iv: "6bbd4a1ce44cb4583ff70bb8450cfd93"
server_key_version: "encryptionkey@0"
I20260923 18:08:37.806254 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:37.830273 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:37.835112 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:37.835378 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:37.836225 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:37.838014 1081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:08:37.843767 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:08:37.844164 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260923 18:08:37.844308 1025 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:08:37.847710 1025 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:08:37.847817 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20260923 18:08:37.848322 1081 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Bootstrap starting.
I20260923 18:08:37.864015 1025 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:41739
I20260923 18:08:37.864224 1188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:41739 every 8 connection(s)
I20260923 18:08:37.865744 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:37.871899 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1025
I20260923 18:08:37.872179 1081 log.cc:824] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:37.872370 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:38163
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=43105
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:37.882385 1081 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:08:37.882953 1081 tablet_bootstrap.cc:492] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Bootstrap complete.
I20260923 18:08:37.884148 1081 ts_tablet_manager.cc:1397] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.000s
I20260923 18:08:37.887921 1081 raft_consensus.cc:348] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.888774 1081 raft_consensus.cc:729] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2b445b086e4ae79d0f7980251595db, State: Initialized, Role: FOLLOWER
I20260923 18:08:37.889367 1081 consensus_queue.cc:260] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:37.890471 1081 ts_tablet_manager.cc:1428] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260923 18:08:37.994139 1191 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:08:37.869296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:40769)
0923 18:08:37.870054 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.870127 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:37.871210 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:37.993056 (+121846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":217,"threads_started":1}
W20260923 18:08:37.994805 1189 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45539 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45539: Client connection negotiation failed: client connection to 127.18.44.254:45539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:08:37.995688 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.995392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:55017)
0923 18:08:37.995522 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.995608 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 18:08:37.996436 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:37.996206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:44755)
0923 18:08:37.996324 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:37.996376 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 18:08:37.996624 1189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:08:38.045625 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:38.046191 1193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:38.046280 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:38.052062 1193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:38.052172 1193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:38.052263 1193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:38.059511 1193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:38163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=43105
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:38.060737 1193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:38.062438 1193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:38.075003 1200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:38.075105 1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:38.075134 1202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:38.076604 1193 server_base.cc:1034] running on GCE node
I20260923 18:08:38.077354 1193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:38.078291 1193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:38.079517 1193 hybrid_clock.cc:648] HybridClock initialized: now 1790186918079480 us; error 44 us; skew 500 ppm
Sep 23 18:08:38 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186918, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:38.083168 1193 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:38.084421 1193 webserver.cc:460] Webserver started at http://127.18.44.194:43105/ using document root <none> and password file <none>
I20260923 18:08:38.085016 1193 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:38.085095 1193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:38.090484 1193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:08:38.093302 1209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.095062 1193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:08:38.095230 1193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "d36fde91c2d4471f8642db21358dccff"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "24e32b365902d3040cadae28257ebc1e"
server_key_iv: "82ff9a81d69b303d5c1266284ee0094d"
server_key_version: "encryptionkey@0"
I20260923 18:08:38.095944 1193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:38.116997 1193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:38.120880 1193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:38.121079 1193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:38.121776 1193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:38.122706 1193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:38.122756 1193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.122797 1193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:38.122812 1193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.133029 1193 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:38163
I20260923 18:08:38.133126 1322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:38163 every 8 connection(s)
I20260923 18:08:38.134150 1193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:38.137130 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1193
I20260923 18:08:38.137596 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:43673
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:45539
--webserver_port=41157
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:08:38.139324 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.136808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:53121)
0923 18:08:38.137582 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.137752 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":188,"threads_started":1}
W20260923 18:08:38.139991 1323 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45539 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45539: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
I20260923 18:08:38.140785 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.140507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:44133)
0923 18:08:38.140659 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.140717 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 18:08:38.141673 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.141339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:43399)
0923 18:08:38.141545 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.141612 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 18:08:38.141870 1323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:08:38.278904 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:38.279309 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:38.279384 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:38.283960 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:38.284066 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:38.284243 1326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:38.292500 1326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=41157
--tserver_master_addrs=127.18.44.254:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:38.293843 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:38.295406 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:38.307893 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:38.308084 1326 server_base.cc:1034] running on GCE node
W20260923 18:08:38.308008 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:38.307893 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:38.308748 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:38.309513 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:38.310731 1326 hybrid_clock.cc:648] HybridClock initialized: now 1790186918310694 us; error 55 us; skew 500 ppm
Sep 23 18:08:38 dist-test-slave-rvjj krb5kdc[32342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186918, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:38.314157 1326 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:38.315282 1326 webserver.cc:460] Webserver started at http://127.18.44.195:41157/ using document root <none> and password file <none>
I20260923 18:08:38.315904 1326 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:38.315987 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:38.320501 1326 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:08:38.323498 1341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.325404 1326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:38.325603 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7"
format_stamp: "Formatted at 2026-09-23 18:08:35 on dist-test-slave-rvjj"
server_key: "5bf838db3331bda4de96609b77540d33"
server_key_iv: "fded2e758afcd5a3a18f376af8c1ad30"
server_key_version: "encryptionkey@0"
I20260923 18:08:38.326237 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:38.342209 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:38.346606 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:38.346920 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:38.348104 1326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:38.349197 1326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:38.349272 1326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.349335 1326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:38.349363 1326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:38.359089 1326 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:43673
I20260923 18:08:38.359150 1454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:43673 every 8 connection(s)
I20260923 18:08:38.360266 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:38.365056 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.362969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:41989)
0923 18:08:38.363629 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.363774 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":177,"threads_started":1}
W20260923 18:08:38.365739 1455 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45539 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45539: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
I20260923 18:08:38.366573 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.366204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:49393)
0923 18:08:38.366378 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.366437 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 18:08:38.366741 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1326
I20260923 18:08:38.367422 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.367170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:46457)
0923 18:08:38.367311 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.367361 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 18:08:38.367633 1455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:08:38.998188 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:38.997361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:35567)
0923 18:08:38.997915 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:38.998030 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":92,"threads_started":1}
I20260923 18:08:39.143603 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.142697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:41713)
0923 18:08:39.143181 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:39.143430 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":100,"threads_started":1}
I20260923 18:08:39.177728 1461 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:39.178009 1461 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:39.179194 1461 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163), 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
Sep 23 18:08:39 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.194@KRBTEST.COM
Sep 23 18:08:39 dist-test-slave-rvjj krb5kdc[32342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186917, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.195@KRBTEST.COM
I20260923 18:08:39.200129 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.179924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:45377 (local address 127.18.44.195:43673)
0923 18:08:39.180677 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:39.180697 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:39.180890 (+ 193us) server_negotiation.cc:407] Connection header received
0923 18:08:39.182260 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:39.182272 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:39.182893 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:39.183416 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:39.184576 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.186027 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.187606 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.188425 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.188623 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:39.193073 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:39.193087 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:39.193119 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:39.193202 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:39.196121 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:39.196987 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:39.196996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:39.197002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:39.197233 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:39.197715 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:39.197719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:39.197720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:39.198344 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:39.198516 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:39.199525 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0923 18:08:39.199910 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":153,"threads_started":1}
I20260923 18:08:39.200129 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.179952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:34797 (local address 127.18.44.194:38163)
0923 18:08:39.180583 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:39.180614 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:39.180673 (+ 59us) server_negotiation.cc:407] Connection header received
0923 18:08:39.181565 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:39.181574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:39.182088 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:39.182427 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:39.184666 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.185809 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.187871 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.188422 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.188625 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:39.193177 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:39.193185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:39.193208 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:39.193258 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:39.195480 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:39.196532 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:39.196541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:39.196551 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:39.196743 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:39.197636 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:39.197640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:39.197642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:39.198344 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:39.198520 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:39.199566 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0923 18:08:39.199909 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":188,"threads_started":1}
I20260923 18:08:39.200142 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.179737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:38163 (local address 127.18.44.193:34797)
0923 18:08:39.180090 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:39.180112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:39.180268 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:39.182745 (+ 2477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:39.182754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:39.183588 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:39.184299 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:39.184329 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.186084 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.186089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:39.187683 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:39.187696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.188612 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.188616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:39.188893 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:39.189836 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:39.189896 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:39.192516 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:39.195727 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:39.195740 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:39.195774 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:39.196286 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:39.196885 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:39.196889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:39.196927 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:39.197372 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:39.198585 (+ 1213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:39.198588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:39.199039 (+ 451us) client_negotiation.cc:769] Sending connection context
0923 18:08:39.199316 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 18:08:39.199912 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":589,"spinlock_wait_cycles":40320}
I20260923 18:08:39.200155 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.179737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:43673 (local address 127.18.44.193:45377)
0923 18:08:39.180735 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:39.180752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:39.180906 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:39.183417 (+ 2511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:39.183421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:39.183747 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:39.184298 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:39.184328 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.186362 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.186366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:39.187416 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:39.187431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:39.188612 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:39.188616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:39.189037 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:39.189832 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:39.189862 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:39.192516 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:39.196388 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:39.196395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:39.196398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:39.196744 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:39.197422 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:39.197425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:39.197428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:39.197592 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:39.198576 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:39.198586 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:39.199091 (+ 505us) client_negotiation.cc:769] Sending connection context
0923 18:08:39.199356 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 18:08:39.199839 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":808,"spinlock_wait_cycles":27264,"thread_start_us":101,"threads_started":1}
W20260923 18:08:39.204200 1075 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
W20260923 18:08:39.204504 1078 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
I20260923 18:08:39.204582 1078 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db; no voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff
I20260923 18:08:39.205169 1461 raft_consensus.cc:2726] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:08:39.369148 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.368436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:35675)
0923 18:08:39.368918 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:39.369021 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260923 18:08:40.001142 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:39.999430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:40747)
0923 18:08:40.000445 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:40.000580 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":210,"threads_started":1}
I20260923 18:08:40.145144 1467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:40.144669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:48051)
0923 18:08:40.144978 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:40.145044 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260923 18:08:40.371224 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:40.370453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:56925)
0923 18:08:40.370797 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:40.370871 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
I20260923 18:08:41.003230 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:41.002376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:47063)
0923 18:08:41.002858 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:41.003067 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":150,"threads_started":1}
I20260923 18:08:41.141980 1470 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:41.142179 1470 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:41.142804 1470 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163), 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
W20260923 18:08:41.144037 1075 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
W20260923 18:08:41.144182 1078 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
I20260923 18:08:41.144352 1078 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db; no voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff
I20260923 18:08:41.144559 1470 raft_consensus.cc:2726] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:08:41.146816 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:41.146040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:53059)
0923 18:08:41.146563 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:41.146673 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":125,"threads_started":1}
I20260923 18:08:41.373067 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:41.372509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:48845)
0923 18:08:41.372876 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:41.372956 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260923 18:08:42.005394 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:42.004749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:38625)
0923 18:08:42.005190 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:42.005283 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260923 18:08:42.148819 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:42.148209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:34049)
0923 18:08:42.148614 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:42.148681 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
I20260923 18:08:42.374853 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:42.374237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:34579)
0923 18:08:42.374660 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:42.374751 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":122,"threads_started":1}
I20260923 18:08:43.007459 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:43.006632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:59695)
0923 18:08:43.007160 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:43.007316 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":81,"threads_started":1}
I20260923 18:08:43.150852 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:43.150158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:45625)
0923 18:08:43.150579 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:43.150681 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":152,"threads_started":1}
I20260923 18:08:43.376592 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:43.376029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:50803)
0923 18:08:43.376352 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:43.376420 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260923 18:08:44.009397 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:44.008750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:44747)
0923 18:08:44.009198 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:44.009280 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":138,"threads_started":1}
I20260923 18:08:44.152890 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:44.152053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:58621)
0923 18:08:44.152554 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:44.152728 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":101,"threads_started":1}
I20260923 18:08:44.328845 1481 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:44.329108 1481 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:44.329913 1481 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163), 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
W20260923 18:08:44.331187 1078 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
W20260923 18:08:44.331540 1075 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
I20260923 18:08:44.331682 1075 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db; no voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff
I20260923 18:08:44.332152 1481 raft_consensus.cc:2726] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:08:44.378724 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:44.377969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:48687)
0923 18:08:44.378416 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:44.378528 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":117,"threads_started":1}
I20260923 18:08:45.011216 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:45.010649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:50303)
0923 18:08:45.010990 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:45.011114 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":87,"threads_started":1}
I20260923 18:08:45.155055 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:45.154222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:57509)
0923 18:08:45.154796 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:45.154905 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":160,"threads_started":1}
I20260923 18:08:45.380558 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:45.379966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:54485)
0923 18:08:45.380388 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:45.380448 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":101,"threads_started":1}
I20260923 18:08:46.013299 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:46.012629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:34017)
0923 18:08:46.013138 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:46.013201 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":96,"threads_started":1}
I20260923 18:08:46.156872 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:46.156164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:48477)
0923 18:08:46.156602 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:46.156734 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
I20260923 18:08:46.382473 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:46.381742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:38125)
0923 18:08:46.382175 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:46.382356 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":146,"threads_started":1}
I20260923 18:08:46.594477 1489 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:46.594795 1489 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:46.595449 1489 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163), 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
W20260923 18:08:46.596669 1075 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
W20260923 18:08:46.596904 1078 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
I20260923 18:08:46.597002 1078 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db; no voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff
I20260923 18:08:46.597199 1489 raft_consensus.cc:2726] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:08:47.015082 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:47.014405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:49297)
0923 18:08:47.014841 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:47.014943 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":91,"threads_started":1}
I20260923 18:08:47.158962 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:47.158076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:40123)
0923 18:08:47.158441 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:47.158780 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260923 18:08:47.384598 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:47.383878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:33191)
0923 18:08:47.384305 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:47.384409 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":120,"threads_started":1}
I20260923 18:08:48.016876 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:48.016237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:43051)
0923 18:08:48.016648 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:48.016719 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":94,"threads_started":1}
I20260923 18:08:48.160889 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:48.160296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:47347)
0923 18:08:48.160675 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:48.160768 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260923 18:08:48.386389 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:48.385781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:36031)
0923 18:08:48.386205 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:48.386287 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":114,"threads_started":1}
I20260923 18:08:49.018476 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:49.017793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:42369)
0923 18:08:49.018233 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:49.018340 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":116,"threads_started":1}
I20260923 18:08:49.162434 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:49.161919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:40615)
0923 18:08:49.162236 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:49.162307 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260923 18:08:49.388290 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:49.387717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:42169)
0923 18:08:49.388069 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:49.388157 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260923 18:08:50.020171 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:50.019657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:33425)
0923 18:08:50.019995 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:50.020065 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
I20260923 18:08:50.163990 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:50.163492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:52817)
0923 18:08:50.163776 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:50.163886 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260923 18:08:50.389918 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:50.389441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:40197)
0923 18:08:50.389738 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:50.389805 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260923 18:08:50.919729 1502 raft_consensus.cc:482] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:50.919963 1502 raft_consensus.cc:504] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d36fde91c2d4471f8642db21358dccff" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 38163 } } peers { permanent_uuid: "3cf3bd0f3cdd4d9eafb26b507ee75bd7" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 43673 } } peers { permanent_uuid: "4e2b445b086e4ae79d0f7980251595db" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41739 } }
I20260923 18:08:50.920411 1502 leader_election.cc:288] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163), 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673)
W20260923 18:08:50.921396 1078 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3cf3bd0f3cdd4d9eafb26b507ee75bd7 (127.18.44.195:43673): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
W20260923 18:08:50.921528 1075 leader_election.cc:341] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d36fde91c2d4471f8642db21358dccff (127.18.44.194:38163): Not found: Tablet not found: 6c185515656849a380217430c2d5f0ff
I20260923 18:08:50.921598 1075 leader_election.cc:302] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e2b445b086e4ae79d0f7980251595db; no voters: 3cf3bd0f3cdd4d9eafb26b507ee75bd7, d36fde91c2d4471f8642db21358dccff
I20260923 18:08:50.921747 1502 raft_consensus.cc:2726] T 6c185515656849a380217430c2d5f0ff P 4e2b445b086e4ae79d0f7980251595db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:08:51.022300 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:51.021568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:37603)
0923 18:08:51.021995 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:51.022099 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260923 18:08:51.165515 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:51.164798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:46857)
0923 18:08:51.165259 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:51.165369 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":100,"threads_started":1}
I20260923 18:08:51.391894 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:51.391160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:50353)
0923 18:08:51.391613 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:51.391749 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":155,"threads_started":1}
I20260923 18:08:52.024358 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:52.023691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:36983)
0923 18:08:52.024155 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:52.024248 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":121,"threads_started":1}
I20260923 18:08:52.167500 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:52.166829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:34003)
0923 18:08:52.167221 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:52.167345 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260923 18:08:52.394124 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:52.393360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.195:57195)
0923 18:08:52.393878 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:52.393984 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":126,"threads_started":1}
I20260923 18:08:53.026433 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:53.025806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.193:33191)
0923 18:08:53.026230 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:53.026318 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":114,"threads_started":1}
I20260923 18:08:53.169420 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:53.168821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45539 (local address 127.18.44.194:32837)
0923 18:08:53.169218 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:53.169309 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260923 18:08:53.372368 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1025
I20260923 18:08:53.378423 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1193
I20260923 18:08:53.385398 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1326
2026-09-23T18:08:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20209 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:08:53 dist-test-slave-rvjj krb5kdc[1513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:08:53 dist-test-slave-rvjj krb5kdc[1513](info): set up 2 sockets
Sep 23 18:08:53 dist-test-slave-rvjj krb5kdc[1513](info): commencing operation
krb5kdc: starting...
W20260923 18:08:54.380682 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:08:54 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:08:54.526522 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:54.657918 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:54.658238 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:54.658294 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:54.662578 1526 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:08:54.662667 1526 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:08:54.662688 1526 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:08:54.662706 1526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:08:54.662730 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:54.662755 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:54.662776 1526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:08:54.662801 1526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:08:54.668754 1526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:54.669938 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:54.671020 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:54.681496 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:54.681592 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:54.681669 1526 server_base.cc:1034] running on GCE node
W20260923 18:08:54.681383 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:54.682443 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:54.683831 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:54.685094 1526 hybrid_clock.cc:648] HybridClock initialized: now 1790186934685042 us; error 47 us; skew 500 ppm
Sep 23 18:08:54 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186934, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:54.688256 1526 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:08:54.689386 1526 webserver.cc:460] Webserver started at http://127.18.44.254:43609/ using document root <none> and password file <none>
I20260923 18:08:54.689980 1526 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:54.690054 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:54.690266 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:54.692526 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "554e8cf9d0584aa591707f155a898a57"
format_stamp: "Formatted at 2026-09-23 18:08:54 on dist-test-slave-rvjj"
server_key: "a8bb27bb80aca8a9da99e4979cbb44c9"
server_key_iv: "8eeb70f064d42673ff28cb814f09b890"
server_key_version: "encryptionkey@0"
I20260923 18:08:54.693150 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "554e8cf9d0584aa591707f155a898a57"
format_stamp: "Formatted at 2026-09-23 18:08:54 on dist-test-slave-rvjj"
server_key: "a8bb27bb80aca8a9da99e4979cbb44c9"
server_key_iv: "8eeb70f064d42673ff28cb814f09b890"
server_key_version: "encryptionkey@0"
I20260923 18:08:54.697515 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:08:54.700350 1541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:54.701953 1526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:08:54.702127 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "554e8cf9d0584aa591707f155a898a57"
format_stamp: "Formatted at 2026-09-23 18:08:54 on dist-test-slave-rvjj"
server_key: "a8bb27bb80aca8a9da99e4979cbb44c9"
server_key_iv: "8eeb70f064d42673ff28cb814f09b890"
server_key_version: "encryptionkey@0"
I20260923 18:08:54.702263 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:54.737679 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:54.741379 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:54.741609 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:54.748814 1526 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45651
I20260923 18:08:54.749161 1593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45651 every 8 connection(s)
I20260923 18:08:54.750114 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:08:54.753057 1594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:54.755782 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1526
I20260923 18:08:54.755904 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:08:54.756196 18611 external_mini_cluster.cc:1428] Setting key 82910d91aa868283f0b3cebdb6916ee3
I20260923 18:08:54.760195 1594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap starting.
I20260923 18:08:54.763329 1594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:54.764248 1594 log.cc:824] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:54.766681 1594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: No bootstrap required, opened a new log
Sep 23 18:08:54 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:54.770385 1594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:08:54.770747 1594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:54.770817 1594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Initialized, Role: FOLLOWER
I20260923 18:08:54.771373 1594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:08:54.771525 1594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:08:54.771597 1594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:08:54.771714 1594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:54.772722 1594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:08:54.773090 1594 leader_election.cc:302] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554e8cf9d0584aa591707f155a898a57; no voters:
I20260923 18:08:54.773440 1594 leader_election.cc:288] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:08:54.773561 1599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:54.773729 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:54.757737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.18.44.254:45651)
0923 18:08:54.758268 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:54.758290 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:54.758339 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:08:54.759291 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:54.759320 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:54.759914 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:54.760405 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:54.761285 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:54.762837 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:54.764023 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:54.764752 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:54.764927 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:54.767798 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:54.767805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:54.767821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:54.767853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:54.770183 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:54.770842 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:54.770849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:54.770855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:54.771048 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:54.771393 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:54.771396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:54.771398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:54.771922 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:54.772134 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:54.772442 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 18:08:54.772748 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":150,"threads_started":1}
I20260923 18:08:54.773818 1599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 LEADER]: Becoming Leader. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Running, Role: LEADER
I20260923 18:08:54.774339 1599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:08:54.774757 1594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:08:54.775689 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554e8cf9d0584aa591707f155a898a57. Latest consensus state: current_term: 1 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:08:54.775820 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
I20260923 18:08:54.776556 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:08:54.776731 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
W20260923 18:08:54.779665 1614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:08:54.779740 1614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:08:54.779855 1608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:08:54.780575 1608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:08:54.787410 1608 catalog_manager.cc:1347] Generated new cluster ID: cca55a528c9a46c3853400200eb2eac4
I20260923 18:08:54.787501 1608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:08:54.799551 1608 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:08:54.801005 1608 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:08:54.817258 1608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Generated new TSK 0
I20260923 18:08:54.818259 1608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:08:54.857344 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:54.983632 1620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:54.983945 1620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:54.984001 1620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:54.988425 1620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:54.988524 1620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:54.988590 1620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:08:54.994463 1620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:54.995659 1620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:54.996783 1620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:55.011783 1626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:55.011808 1625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:55.011855 1620 server_base.cc:1034] running on GCE node
W20260923 18:08:55.011803 1628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:55.012614 1620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:55.013377 1620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:55.014596 1620 hybrid_clock.cc:648] HybridClock initialized: now 1790186935014577 us; error 41 us; skew 500 ppm
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:55.024554 1620 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:08:55.025914 1620 webserver.cc:460] Webserver started at http://127.18.44.193:37559/ using document root <none> and password file <none>
I20260923 18:08:55.026911 1620 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:55.027029 1620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:55.027403 1620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:55.030153 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "61f8acb5a6654709a0c5b2d69960deb0"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "b9b2560afc2c1b6c07a6460902510335"
server_key_iv: "ac38ab9294c21e69e88bb9f7120fd411"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.030848 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "61f8acb5a6654709a0c5b2d69960deb0"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "b9b2560afc2c1b6c07a6460902510335"
server_key_iv: "ac38ab9294c21e69e88bb9f7120fd411"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.035643 1620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 18:08:55.038888 1635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.041152 1620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:55.041326 1620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "61f8acb5a6654709a0c5b2d69960deb0"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "b9b2560afc2c1b6c07a6460902510335"
server_key_iv: "ac38ab9294c21e69e88bb9f7120fd411"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.041648 1620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:55.085250 1620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:55.088832 1620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:55.089030 1620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:55.089661 1620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:55.090587 1620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:55.090662 1620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.090723 1620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:55.090747 1620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.101020 1620 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:35355
I20260923 18:08:55.101159 1748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:35355 every 8 connection(s)
I20260923 18:08:55.102067 1620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:08:55.106417 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1620
I20260923 18:08:55.106611 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:08:55.107223 18611 external_mini_cluster.cc:1428] Setting key 93987c20d60631462d8c6c23287b291f
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:08:55.125604 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.105120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:47193 (local address 127.18.44.254:45651)
0923 18:08:55.105452 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.105457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.106878 (+ 1421us) server_negotiation.cc:407] Connection header received
0923 18:08:55.109249 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.109254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.109336 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.109440 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:55.111562 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.112366 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.114055 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.114353 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.114480 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.118373 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:55.118377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:55.118380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:55.118421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:55.120439 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.121416 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.121420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.121421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.121539 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.122001 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.122004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.122006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.122654 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:55.123220 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.125223 (+ 2003us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.125428 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260923 18:08:55.127526 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.105549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:47193)
0923 18:08:55.106464 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.106610 (+ 146us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.108765 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.109844 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.109856 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.110442 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:55.111240 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.111327 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.112617 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.112622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.113853 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.113871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.114579 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.114583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.114838 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.115899 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:55.115924 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:55.118039 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:55.120659 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.120668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.120693 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.121262 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.121646 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.121650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.121652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.121880 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.123482 (+ 1602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:55.123492 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:55.124125 (+ 633us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.125065 (+ 940us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.125650 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":160,"threads_started":1}
I20260923 18:08:55.129738 1749 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:08:55.130082 1749 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:55.130733 1749 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:08:55.133158 1558 ts_manager.cc:194] Registered new tserver with Master: 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355)
I20260923 18:08:55.134531 1558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:47193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:08:55.153550 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:55.273303 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:55.273620 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:55.273665 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:55.278326 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:55.278417 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:55.278486 1754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:08:55.285140 1754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:55.286315 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:55.287444 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:55.299109 1759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:55.299070 1760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:55.299387 1754 server_base.cc:1034] running on GCE node
W20260923 18:08:55.299561 1762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:55.300258 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:55.301257 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:55.302489 1754 hybrid_clock.cc:648] HybridClock initialized: now 1790186935302448 us; error 58 us; skew 500 ppm
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:55.307363 1754 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:08:55.309053 1754 webserver.cc:460] Webserver started at http://127.18.44.194:38313/ using document root <none> and password file <none>
I20260923 18:08:55.310552 1754 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:55.310719 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:55.311026 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:55.314316 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "c96d5f2301d0496fbfc771d580796061"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "454cd99afeaa0d99e1c9f97277da4d35"
server_key_iv: "91f68cf3f59df5d4e56034fd52bb3238"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.315137 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "c96d5f2301d0496fbfc771d580796061"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "454cd99afeaa0d99e1c9f97277da4d35"
server_key_iv: "91f68cf3f59df5d4e56034fd52bb3238"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.320978 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:08:55.326769 1769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.328680 1754 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:08:55.328861 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "c96d5f2301d0496fbfc771d580796061"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "454cd99afeaa0d99e1c9f97277da4d35"
server_key_iv: "91f68cf3f59df5d4e56034fd52bb3238"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.329020 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:55.345726 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:55.350940 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:55.351179 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:55.351962 1754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:55.353760 1754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:55.353858 1754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.353937 1754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:55.353960 1754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.368132 1754 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:46833
I20260923 18:08:55.368153 1882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:46833 every 8 connection(s)
I20260923 18:08:55.369645 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:08:55.373522 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1754
I20260923 18:08:55.373669 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:08:55.374038 18611 external_mini_cluster.cc:1428] Setting key 6f66f3b0d48027b3cbe3d3585df0671f
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:55.399551 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.372613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:48373 (local address 127.18.44.254:45651)
0923 18:08:55.372755 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.372759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.375326 (+ 2567us) server_negotiation.cc:407] Connection header received
0923 18:08:55.377463 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.377466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.377532 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.377605 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:55.379982 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.380834 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.383439 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.383752 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.383926 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.394222 (+ 10296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:55.394228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:55.394232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:55.394284 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:55.396392 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.397311 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.397316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.397317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.397430 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.397911 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.397914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.397916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.398191 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:55.398298 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.399051 (+ 753us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.399323 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:08:55.400760 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.373062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:48373)
0923 18:08:55.375073 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.375162 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.376886 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.377997 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.378011 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.378776 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:55.379643 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.379677 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.381091 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.381095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.383013 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.383049 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.384041 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.384048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.384307 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.385091 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:55.385113 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:55.393787 (+ 8674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:55.396646 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.396656 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.396682 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.397136 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.397559 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.397563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.397566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.397787 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.398323 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:55.398334 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:55.398730 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.398963 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.399517 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":396,"threads_started":1}
I20260923 18:08:55.402710 1883 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:08:55.403505 1883 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:55.404510 1883 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:08:55.407053 1558 ts_manager.cc:194] Registered new tserver with Master: c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833)
I20260923 18:08:55.407793 1558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:48373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:08:55.418330 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:08:55.541994 1888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:08:55.542325 1888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:08:55.542371 1888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:08:55.546954 1888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:08:55.547061 1888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:08:55.547138 1888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:08:55.553294 1888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:08:55.554541 1888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:08:55.555673 1888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:08:55.567421 1896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:55.567517 1893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:08:55.567421 1894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:08:55.567840 1888 server_base.cc:1034] running on GCE node
I20260923 18:08:55.568395 1888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:08:55.569141 1888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:08:55.570521 1888 hybrid_clock.cc:648] HybridClock initialized: now 1790186935570475 us; error 46 us; skew 500 ppm
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:08:55.575971 1888 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:08:55.577239 1888 webserver.cc:460] Webserver started at http://127.18.44.195:34261/ using document root <none> and password file <none>
I20260923 18:08:55.577903 1888 fs_manager.cc:359] Metadata directory not provided
I20260923 18:08:55.577986 1888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:08:55.578208 1888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:08:55.580606 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "63ba20486c904e339ae0c9ce7511fcd3"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "8ba3a9523d4d86dd4a384ac986ddcfd7"
server_key_iv: "db97c0830ae1da784a058ff08371934c"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.581243 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "63ba20486c904e339ae0c9ce7511fcd3"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "8ba3a9523d4d86dd4a384ac986ddcfd7"
server_key_iv: "db97c0830ae1da784a058ff08371934c"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.586000 1888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:08:55.589609 1903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.591498 1888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 18:08:55.591665 1888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "63ba20486c904e339ae0c9ce7511fcd3"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "8ba3a9523d4d86dd4a384ac986ddcfd7"
server_key_iv: "db97c0830ae1da784a058ff08371934c"
server_key_version: "encryptionkey@0"
I20260923 18:08:55.591804 1888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:08:55.607180 1888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:08:55.610657 1888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:08:55.610848 1888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:08:55.611544 1888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:08:55.612521 1888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:08:55.612578 1888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.612622 1888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:08:55.612636 1888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:08:55.624926 1888 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40593
I20260923 18:08:55.625047 2016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40593 every 8 connection(s)
I20260923 18:08:55.626063 1888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:08:55.628057 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 1888
I20260923 18:08:55.628212 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:08:55.628607 18611 external_mini_cluster.cc:1428] Setting key a18983781767acf7601260e3acf7e5fd
Sep 23 18:08:55 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186935, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:08:55.652115 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.628678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50607 (local address 127.18.44.254:45651)
0923 18:08:55.628864 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.628870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.629962 (+ 1092us) server_negotiation.cc:407] Connection header received
0923 18:08:55.632569 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.632573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.632669 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.632748 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:55.635523 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.636269 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.638514 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.638903 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.639104 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.644515 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:55.644520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:55.644524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:55.644581 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:55.647884 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.648927 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.648939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.648941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.649105 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.649883 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.649887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.649888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.650365 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:55.650499 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.651768 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.651951 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 18:08:55.653017 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.629113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:50607)
0923 18:08:55.629802 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.629860 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.632201 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.633100 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.633113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.633716 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:08:55.635039 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.635082 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.636541 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.636545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.638106 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.638132 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.639497 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.639513 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.639794 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.640977 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:08:55.641005 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:08:55.643928 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:08:55.648118 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.648126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.648145 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.648693 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.649311 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:08:55.649314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:08:55.649315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:08:55.649692 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:08:55.650610 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:08:55.650617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:08:55.651167 (+ 550us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.651520 (+ 353us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.651914 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":192,"threads_started":1}
I20260923 18:08:55.655131 2017 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:08:55.655570 2017 heartbeater.cc:464] Registering TS with master...
I20260923 18:08:55.656212 2017 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:08:55.658401 1557 ts_manager.cc:194] Registered new tserver with Master: 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:08:55.659242 1557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:50607
I20260923 18:08:55.659454 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:08:55.672716 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.662527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.18.44.254:45651)
0923 18:08:55.662763 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.662770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.662874 (+ 104us) server_negotiation.cc:407] Connection header received
0923 18:08:55.663100 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.663104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.663162 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.663244 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:08:55.664151 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.664751 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.665745 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.666019 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.666133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.668261 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:08:55.668265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:08:55.668267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:08:55.668313 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:08:55.670201 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.670817 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.670821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.670823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.670922 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:08:55.671207 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:08:55.671211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:08:55.671212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:08:55.671597 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:08:55.671754 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.672310 (+ 556us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.672499 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:08:55.679371 1557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:08:55.682848 1557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:08:55.703477 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.697305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35355 (local address 127.0.0.1:48874)
0923 18:08:55.697697 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.697710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.697830 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.699009 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.699013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.699032 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:55.699196 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.699205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.700460 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.700464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.701699 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.701712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.702777 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.702781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.702920 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.702971 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.703101 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.703297 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260923 18:08:55.703488 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.697700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.18.44.193:35355)
0923 18:08:55.698530 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.698538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.698573 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:08:55.698654 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.698659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.698846 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.699002 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:55.699398 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.700316 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.701864 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.702623 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.702731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.702905 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.703152 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.703307 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":82,"threads_started":1}
I20260923 18:08:55.704295 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.695854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46833 (local address 127.0.0.1:59618)
0923 18:08:55.696719 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.696819 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.696999 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.698181 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.698189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.698218 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:55.698445 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.698456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.700153 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.700157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.702098 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.702110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.703527 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.703531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.703825 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.703885 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.703954 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.704093 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":439,"thread_start_us":125,"threads_started":1}
I20260923 18:08:55.704653 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.698182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.18.44.195:40593)
0923 18:08:55.698665 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.698679 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.698705 (+ 26us) server_negotiation.cc:407] Connection header received
0923 18:08:55.698792 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.698798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.699044 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.699230 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:55.699477 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.700526 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.702181 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.703334 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.703472 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.703694 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.704120 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.704286 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":156,"threads_started":1}
I20260923 18:08:55.704730 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.697581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40593 (local address 127.0.0.1:35432)
0923 18:08:55.698131 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.698147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.698279 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.699224 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.699226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.699244 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:55.699366 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.699372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.700695 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.700700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.702000 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.702015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.703765 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.703769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.703907 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.703928 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.703996 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.704586 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":31,"spinlock_wait_cycles":1280,"thread_start_us":123,"threads_started":1}
I20260923 18:08:55.705274 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.696136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.18.44.194:46833)
0923 18:08:55.697449 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.697466 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.697500 (+ 34us) server_negotiation.cc:407] Connection header received
0923 18:08:55.697600 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.697606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.697936 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.698197 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:55.698604 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.699949 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.702268 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.703334 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.704096 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.704298 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.704377 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.705070 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":159,"threads_started":1}
I20260923 18:08:55.707754 1683 tablet_service.cc:1467] Processing CreateTablet for tablet 011b87260cc1482d87bcb00ce011fd81 (DEFAULT_TABLE table=test-table [id=b1780f5ce57a46dc9196289a86ca7757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:55.709285 1683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011b87260cc1482d87bcb00ce011fd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:55.709692 1817 tablet_service.cc:1467] Processing CreateTablet for tablet 011b87260cc1482d87bcb00ce011fd81 (DEFAULT_TABLE table=test-table [id=b1780f5ce57a46dc9196289a86ca7757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:55.709614 1951 tablet_service.cc:1467] Processing CreateTablet for tablet 011b87260cc1482d87bcb00ce011fd81 (DEFAULT_TABLE table=test-table [id=b1780f5ce57a46dc9196289a86ca7757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:08:55.711117 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011b87260cc1482d87bcb00ce011fd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:55.711640 1951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 011b87260cc1482d87bcb00ce011fd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:08:55.719848 2034 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap starting.
I20260923 18:08:55.723610 2035 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Bootstrap starting.
I20260923 18:08:55.724200 2034 tablet_bootstrap.cc:654] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:55.724895 2036 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Bootstrap starting.
I20260923 18:08:55.726424 2034 log.cc:824] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:55.726974 2035 tablet_bootstrap.cc:654] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:55.728824 2035 log.cc:824] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:55.729189 2036 tablet_bootstrap.cc:654] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Neither blocks nor log segments found. Creating new log.
I20260923 18:08:55.730410 2036 log.cc:824] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Log is configured to *not* fsync() on all Append() calls
I20260923 18:08:55.730697 2034 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: No bootstrap required, opened a new log
I20260923 18:08:55.730952 2034 ts_tablet_manager.cc:1397] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 18:08:55.731068 2035 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: No bootstrap required, opened a new log
I20260923 18:08:55.731312 2035 ts_tablet_manager.cc:1397] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 18:08:55.732740 2036 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: No bootstrap required, opened a new log
I20260923 18:08:55.732959 2036 ts_tablet_manager.cc:1397] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 18:08:55.735737 2035 raft_consensus.cc:348] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.736254 2035 raft_consensus.cc:374] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:55.736328 2035 raft_consensus.cc:729] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ba20486c904e339ae0c9ce7511fcd3, State: Initialized, Role: FOLLOWER
I20260923 18:08:55.736986 2035 consensus_queue.cc:260] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.736878 2034 raft_consensus.cc:348] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.737366 2034 raft_consensus.cc:374] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:55.737452 2034 raft_consensus.cc:729] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8acb5a6654709a0c5b2d69960deb0, State: Initialized, Role: FOLLOWER
I20260923 18:08:55.737635 2036 raft_consensus.cc:348] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.737999 2036 raft_consensus.cc:374] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:08:55.738068 2036 raft_consensus.cc:729] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96d5f2301d0496fbfc771d580796061, State: Initialized, Role: FOLLOWER
I20260923 18:08:55.738093 2017 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
I20260923 18:08:55.738125 2034 consensus_queue.cc:260] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.738602 2036 consensus_queue.cc:260] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.739377 1749 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
I20260923 18:08:55.739918 2034 ts_tablet_manager.cc:1428] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent starting tablet: real 0.009s user 0.004s sys 0.007s
I20260923 18:08:55.739856 2036 ts_tablet_manager.cc:1428] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 18:08:55.740600 1883 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
I20260923 18:08:55.742235 2035 ts_tablet_manager.cc:1428] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260923 18:08:55.782203 2042 raft_consensus.cc:482] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:08:55.782461 2042 raft_consensus.cc:504] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.783826 2042 leader_election.cc:288] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:08:55.790369 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.784705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:38821 (local address 127.18.44.195:40593)
0923 18:08:55.784844 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.784848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.784865 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:08:55.784926 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.784928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.784973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.785137 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:55.785548 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.786476 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.788795 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.789566 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.789694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.789742 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.790137 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.790226 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:08:55.790369 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.784581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:56091 (local address 127.18.44.193:35355)
0923 18:08:55.784828 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.784834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.785453 (+ 619us) server_negotiation.cc:407] Connection header received
0923 18:08:55.785556 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.785558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.785607 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.785698 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:08:55.786047 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.786595 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.788770 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.789500 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.789682 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.789742 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.790137 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.790231 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 18:08:55.790392 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.784424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40593 (local address 127.18.44.194:38821)
0923 18:08:55.784578 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.784596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.784742 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.785192 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.785198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.785230 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:55.785358 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.785364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.786806 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.786812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.788400 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.788419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.789731 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.789736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.789946 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.789994 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.790089 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.790202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":175}
I20260923 18:08:55.790400 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.784424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35355 (local address 127.18.44.194:56091)
0923 18:08:55.785322 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.785336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:08:55.785454 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:08:55.785714 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:08:55.785717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:08:55.785735 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:08:55.785901 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.785907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.786806 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.786812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.788400 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:08:55.788419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.789731 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.789736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:08:55.789942 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.789996 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 18:08:55.790089 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 18:08:55.790202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":520,"thread_start_us":189,"threads_started":1}
I20260923 18:08:55.791131 1703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011b87260cc1482d87bcb00ce011fd81" candidate_uuid: "c96d5f2301d0496fbfc771d580796061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f8acb5a6654709a0c5b2d69960deb0" is_pre_election: true
I20260923 18:08:55.791535 1703 raft_consensus.cc:2454] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96d5f2301d0496fbfc771d580796061 in term 0.
I20260923 18:08:55.791855 1971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011b87260cc1482d87bcb00ce011fd81" candidate_uuid: "c96d5f2301d0496fbfc771d580796061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ba20486c904e339ae0c9ce7511fcd3" is_pre_election: true
I20260923 18:08:55.792248 1773 leader_election.cc:302] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0, c96d5f2301d0496fbfc771d580796061; no voters:
I20260923 18:08:55.792356 1971 raft_consensus.cc:2454] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96d5f2301d0496fbfc771d580796061 in term 0.
I20260923 18:08:55.792737 2042 raft_consensus.cc:2781] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:08:55.792909 2042 raft_consensus.cc:482] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:08:55.792979 2042 raft_consensus.cc:3037] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:55.794160 2042 raft_consensus.cc:504] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.794571 2042 leader_election.cc:288] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [CANDIDATE]: Term 1 election: Requested vote from peers 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:08:55.795084 1971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011b87260cc1482d87bcb00ce011fd81" candidate_uuid: "c96d5f2301d0496fbfc771d580796061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ba20486c904e339ae0c9ce7511fcd3"
I20260923 18:08:55.795301 1971 raft_consensus.cc:3037] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:55.795336 1703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "011b87260cc1482d87bcb00ce011fd81" candidate_uuid: "c96d5f2301d0496fbfc771d580796061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f8acb5a6654709a0c5b2d69960deb0"
I20260923 18:08:55.795463 1703 raft_consensus.cc:3037] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:08:55.796782 1703 raft_consensus.cc:2454] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96d5f2301d0496fbfc771d580796061 in term 1.
I20260923 18:08:55.796780 1971 raft_consensus.cc:2454] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96d5f2301d0496fbfc771d580796061 in term 1.
I20260923 18:08:55.797449 1773 leader_election.cc:302] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0, c96d5f2301d0496fbfc771d580796061; no voters:
I20260923 18:08:55.797804 2042 raft_consensus.cc:2781] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:08:55.798095 2042 raft_consensus.cc:686] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 1 LEADER]: Becoming Leader. State: Replica: c96d5f2301d0496fbfc771d580796061, State: Running, Role: LEADER
I20260923 18:08:55.798480 2042 consensus_queue.cc:237] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:08:55.803745 1558 catalog_manager.cc:5495] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 reported cstate change: term changed from 0 to 1, leader changed from <none> to c96d5f2301d0496fbfc771d580796061 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "c96d5f2301d0496fbfc771d580796061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } health_report { overall_health: UNKNOWN } } }
I20260923 18:08:55.844597 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.839235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.18.44.194:46833)
0923 18:08:55.839536 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 18:08:55.839541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:08:55.839559 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:08:55.839674 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:08:55.839677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:08:55.839753 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:08:55.839938 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:08:55.840339 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.840950 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.841986 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:08:55.842252 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:08:55.842441 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:08:55.842829 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:08:55.843656 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:08:55.843928 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 18:08:55.844301 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 18:08:55.844406 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260923 18:08:55.852481 1971 raft_consensus.cc:1261] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Refusing update from remote peer c96d5f2301d0496fbfc771d580796061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:08:55.852666 1703 raft_consensus.cc:1261] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Refusing update from remote peer c96d5f2301d0496fbfc771d580796061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260923 18:08:55.853778 1750 tablet.cc:2369] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:08:55.853622 2044 consensus_queue.cc:1035] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:55.854120 2042 consensus_queue.cc:1035] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:08:55.865177 2049 mvcc.cc:205] Tried to move back new op lower bound from 7332605689242574848 to 7332605689033474048. Current Snapshot: MvccSnapshot[applied={T|T < 7332605689242574848}]
I20260923 18:08:55.866468 2048 mvcc.cc:205] Tried to move back new op lower bound from 7332605689242574848 to 7332605689033474048. Current Snapshot: MvccSnapshot[applied={T|T < 7332605689242574848}]
I20260923 18:08:55.877388 1558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:55.877849 1558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:08:55.881804 1558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Sending DeleteTablet for 3 replicas of tablet 011b87260cc1482d87bcb00ce011fd81
I20260923 18:08:55.883042 1683 tablet_service.cc:1514] Processing DeleteTablet for tablet 011b87260cc1482d87bcb00ce011fd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:48874
I20260923 18:08:55.883163 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1620
I20260923 18:08:55.883402 1951 tablet_service.cc:1514] Processing DeleteTablet for tablet 011b87260cc1482d87bcb00ce011fd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:35432
I20260923 18:08:55.885680 1817 tablet_service.cc:1514] Processing DeleteTablet for tablet 011b87260cc1482d87bcb00ce011fd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:59618
I20260923 18:08:55.885819 2057 tablet_replica.cc:330] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: stopping tablet replica
I20260923 18:08:55.886507 2057 raft_consensus.cc:2229] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:08:55.886992 2057 raft_consensus.cc:2258] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:55.887768 2058 tablet_replica.cc:330] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: stopping tablet replica
I20260923 18:08:55.888224 2058 raft_consensus.cc:2229] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:08:55.888846 2058 raft_consensus.cc:2258] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:08:55.889582 2057 ts_tablet_manager.cc:1905] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:08:55.890506 2058 ts_tablet_manager.cc:1905] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:08:55.892162 1545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:35355 (error 108)
I20260923 18:08:55.893261 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1754
I20260923 18:08:55.893381 2057 ts_tablet_manager.cc:1918] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:08:55.893514 2057 log.cc:1196] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/011b87260cc1482d87bcb00ce011fd81
I20260923 18:08:55.893831 2057 ts_tablet_manager.cc:1939] T 011b87260cc1482d87bcb00ce011fd81 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting consensus metadata
I20260923 18:08:55.895849 1543 catalog_manager.cc:4841] TS 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): tablet 011b87260cc1482d87bcb00ce011fd81 (table test-table [id=b1780f5ce57a46dc9196289a86ca7757]) successfully deleted
I20260923 18:08:55.899825 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1888
I20260923 18:08:55.901480 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.901024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35355 (local address 127.0.0.1:48878)
0923 18:08:55.901222 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.901333 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 18:08:55.901926 1545 catalog_manager.cc:4584] TS 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 011b87260cc1482d87bcb00ce011fd81: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35355: connect: Connection refused (error 111)
I20260923 18:08:55.902783 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:08:55.902502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46833 (local address 127.0.0.1:59628)
0923 18:08:55.902645 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 18:08:55.902705 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 18:08:55.906090 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 1526
I20260923 18:09:15.914858 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--webserver_interface=127.18.44.254
--webserver_port=43609
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:16.034240 2060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:16.034584 2060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:16.034627 2060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:16.039613 2060 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:09:16.039696 2060 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:09:16.039712 2060 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:09:16.039726 2060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:16.039744 2060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:16.039763 2060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:16.039779 2060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:16.039795 2060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:16.045810 2060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:16.047006 2060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:16.048228 2060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:16.064780 2067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.064916 2066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.064780 2069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.065181 2060 server_base.cc:1034] running on GCE node
I20260923 18:09:16.066032 2060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:16.067471 2060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:16.070931 2060 hybrid_clock.cc:648] HybridClock initialized: now 1790186956070830 us; error 108 us; skew 500 ppm
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:16.082347 2060 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:09:16.083747 2060 webserver.cc:460] Webserver started at http://127.18.44.254:43609/ using document root <none> and password file <none>
I20260923 18:09:16.084450 2060 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:16.084550 2060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:16.089645 2060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:09:16.092828 2076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.094758 2060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:09:16.094966 2060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "554e8cf9d0584aa591707f155a898a57"
format_stamp: "Formatted at 2026-09-23 18:08:54 on dist-test-slave-rvjj"
server_key: "a8bb27bb80aca8a9da99e4979cbb44c9"
server_key_iv: "8eeb70f064d42673ff28cb814f09b890"
server_key_version: "encryptionkey@0"
I20260923 18:09:16.095724 2060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:16.108180 2060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:16.112180 2060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:16.112398 2060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:16.120896 2060 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45651
I20260923 18:09:16.121273 2128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45651 every 8 connection(s)
I20260923 18:09:16.122159 2060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:09:16.124459 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2060
I20260923 18:09:16.125072 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:35355
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=37559
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:09:16.129910 2129 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:09:16.132586 2129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap starting.
I20260923 18:09:16.156886 2129 log.cc:824] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:16.169202 2129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:09:16.169728 2129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap complete.
I20260923 18:09:16.174858 2129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:16.175595 2129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Initialized, Role: FOLLOWER
I20260923 18:09:16.176229 2129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:16.176373 2129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:09:16.176457 2129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:09:16.176571 2129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:09:16.179693 2129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:16.180266 2129 leader_election.cc:302] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554e8cf9d0584aa591707f155a898a57; no voters:
I20260923 18:09:16.180727 2129 leader_election.cc:288] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:09:16.180970 2133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:09:16.181394 2133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 LEADER]: Becoming Leader. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Running, Role: LEADER
I20260923 18:09:16.181806 2133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:16.182487 2129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:09:16.183825 2134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:09:16.184018 2134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:16.184368 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554e8cf9d0584aa591707f155a898a57. Latest consensus state: current_term: 2 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:09:16.184463 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:16.189786 2148 catalog_manager.cc:1259] Loaded cluster ID: cca55a528c9a46c3853400200eb2eac4
I20260923 18:09:16.189898 2148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: loading cluster ID for follower catalog manager: success
I20260923 18:09:16.193037 2148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: acquiring CA information for follower catalog manager: success
I20260923 18:09:16.194964 2148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:09:16.195315 2142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:09:16.199621 2142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:09:16.200412 2142 catalog_manager.cc:1259] Loaded cluster ID: cca55a528c9a46c3853400200eb2eac4
I20260923 18:09:16.200492 2142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:09:16.201419 2142 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:09:16.202042 2142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Loaded TSK: 0
I20260923 18:09:16.220412 2142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Generated new TSK 1
I20260923 18:09:16.220852 2142 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:09:16.252151 2131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:16.252458 2131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:16.252523 2131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:16.256831 2131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:16.256919 2131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:16.257012 2131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:09:16.262903 2131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:35355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=37559
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:16.264135 2131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:16.265224 2131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:16.275856 2156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.275818 2159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.276069 2131 server_base.cc:1034] running on GCE node
W20260923 18:09:16.275818 2157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.276793 2131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:16.277504 2131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:16.278772 2131 hybrid_clock.cc:648] HybridClock initialized: now 1790186956278722 us; error 87 us; skew 500 ppm
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:16.282143 2131 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:09:16.283332 2131 webserver.cc:460] Webserver started at http://127.18.44.193:37559/ using document root <none> and password file <none>
I20260923 18:09:16.284097 2131 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:16.284189 2131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:16.288823 2131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:09:16.291756 2166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.293560 2131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:09:16.293720 2131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "61f8acb5a6654709a0c5b2d69960deb0"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "b9b2560afc2c1b6c07a6460902510335"
server_key_iv: "ac38ab9294c21e69e88bb9f7120fd411"
server_key_version: "encryptionkey@0"
I20260923 18:09:16.294320 2131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:16.311233 2131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:16.315164 2131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:16.315399 2131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:16.315994 2131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:16.317154 2173 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:09:16.321449 2131 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:09:16.321534 2131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 18:09:16.321589 2131 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:09:16.323730 2131 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:09:16.323819 2131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260923 18:09:16.324116 2173 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap starting.
I20260923 18:09:16.336014 2131 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:35355
I20260923 18:09:16.336490 2280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:35355 every 8 connection(s)
I20260923 18:09:16.337527 2131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:09:16.344061 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2131
I20260923 18:09:16.344729 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:46833
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=38313
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:09:16.347038 2173 log.cc:824] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:16.357411 2173 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:09:16.357928 2173 tablet_bootstrap.cc:492] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap complete.
I20260923 18:09:16.359441 2173 ts_tablet_manager.cc:1397] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.008s
I20260923 18:09:16.362957 2173 raft_consensus.cc:348] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:09:16.363611 2173 raft_consensus.cc:729] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8acb5a6654709a0c5b2d69960deb0, State: Initialized, Role: FOLLOWER
I20260923 18:09:16.364192 2173 consensus_queue.cc:260] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } }
I20260923 18:09:16.365389 2173 ts_tablet_manager.cc:1428] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:09:16.366843 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.340705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:46427)
0923 18:09:16.341896 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.341980 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.343499 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.346632 (+ 3133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.346644 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.347470 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:16.348423 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.348458 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.350731 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.350736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.352302 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.352320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.353785 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.353790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.354008 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.354863 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:16.354888 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:16.357442 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:16.361292 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.361301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.361319 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.361807 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.362274 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.362277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.362279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.362475 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.363333 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:16.363340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:16.363893 (+ 553us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.364120 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.365766 (+ 1646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":257,"threads_started":1}
I20260923 18:09:16.367931 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.340831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:46427 (local address 127.18.44.254:45651)
0923 18:09:16.341493 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.341516 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.342204 (+ 688us) server_negotiation.cc:407] Connection header received
0923 18:09:16.344523 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.344540 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.345402 (+ 862us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.345928 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.348795 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.350448 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.352515 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.353325 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.353499 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.357954 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.357964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.358041 (+ 77us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.358093 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.361035 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.361964 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.361972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.361977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.362161 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.362589 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.362592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.362594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.363180 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.364494 (+ 1314us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.364679 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.365058 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":172,"threads_started":1}
I20260923 18:09:16.371037 2281 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:09:16.371447 2281 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:16.372269 2281 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:09:16.375962 2090 ts_manager.cc:194] Registered new tserver with Master: 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355)
I20260923 18:09:16.380416 2090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:46427
I20260923 18:09:16.384088 2281 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.193@KRBTEST.COM
I20260923 18:09:16.394654 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.379937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35355 (local address 127.0.0.1:49012)
0923 18:09:16.381037 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.381085 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.381251 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.382105 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.382113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.382652 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:16.382840 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.382850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.384465 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.384469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.385496 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.385513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.386502 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.386507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.386655 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.387417 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:16.387431 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:16.389328 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:16.392230 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.392248 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.392254 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.392703 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.393126 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.393130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.393132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.393238 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.393837 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:16.393843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:16.394010 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.394179 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.394449 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":194,"threads_started":1}
I20260923 18:09:16.394775 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.380042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.18.44.193:35355)
0923 18:09:16.381480 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.381496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.381534 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:09:16.381641 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.381647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.381874 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.382066 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.382991 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.384256 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.385704 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.386330 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.386473 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.389592 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.389609 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.389627 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.389688 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.391998 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.392890 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.392897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.392899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.393017 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.393349 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.393352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.393354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.393700 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.393825 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.394363 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.394589 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":416,"thread_start_us":123,"threads_started":1}
I20260923 18:09:16.396261 2215 tablet_service.cc:1514] Processing DeleteTablet for tablet 011b87260cc1482d87bcb00ce011fd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:08:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:49012
I20260923 18:09:16.397477 2290 tablet_replica.cc:330] stopping tablet replica
I20260923 18:09:16.397945 2290 raft_consensus.cc:2229] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:16.398290 2290 raft_consensus.cc:2258] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:16.399816 2290 ts_tablet_manager.cc:1905] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:16.404762 2290 ts_tablet_manager.cc:1918] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:16.404907 2290 log.cc:1196] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal/wals/011b87260cc1482d87bcb00ce011fd81
I20260923 18:09:16.405154 2290 ts_tablet_manager.cc:1939] T 011b87260cc1482d87bcb00ce011fd81 P 61f8acb5a6654709a0c5b2d69960deb0: Deleting consensus metadata
I20260923 18:09:16.408149 2080 catalog_manager.cc:4841] TS 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355): tablet 011b87260cc1482d87bcb00ce011fd81 (table test-table [id=b1780f5ce57a46dc9196289a86ca7757]) successfully deleted
W20260923 18:09:16.474223 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:16.474537 2285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:16.474581 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:16.478874 2285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:16.478973 2285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:16.479039 2285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:09:16.484859 2285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:46833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=38313
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:16.486014 2285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:16.487044 2285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:16.498235 2298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.498216 2295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.498404 2285 server_base.cc:1034] running on GCE node
W20260923 18:09:16.498245 2296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.499177 2285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:16.500044 2285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:16.501286 2285 hybrid_clock.cc:648] HybridClock initialized: now 1790186956501258 us; error 54 us; skew 500 ppm
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:16.504791 2285 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:09:16.505934 2285 webserver.cc:460] Webserver started at http://127.18.44.194:38313/ using document root <none> and password file <none>
I20260923 18:09:16.506541 2285 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:16.506630 2285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:16.511195 2285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:09:16.514180 2305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.515913 2285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:09:16.516068 2285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "c96d5f2301d0496fbfc771d580796061"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "454cd99afeaa0d99e1c9f97277da4d35"
server_key_iv: "91f68cf3f59df5d4e56034fd52bb3238"
server_key_version: "encryptionkey@0"
I20260923 18:09:16.516753 2285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:16.544138 2285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:16.548166 2285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:16.548385 2285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:16.549055 2285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:16.550381 2312 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260923 18:09:16.556820 2312 ts_tablet_manager.cc:1852] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 18:09:16.557586 2312 ts_tablet_manager.cc:1905] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:16.563853 2312 ts_tablet_manager.cc:1918] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:16.564029 2312 log.cc:1196] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/011b87260cc1482d87bcb00ce011fd81
I20260923 18:09:16.564360 2312 ts_tablet_manager.cc:1939] T 011b87260cc1482d87bcb00ce011fd81 P c96d5f2301d0496fbfc771d580796061: Deleting consensus metadata
I20260923 18:09:16.564730 2285 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 18:09:16.564859 2285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260923 18:09:16.564985 2285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:16.565017 2285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.575410 2285 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:46833
I20260923 18:09:16.575590 2419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:46833 every 8 connection(s)
I20260923 18:09:16.576493 2285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:09:16.585465 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2285
I20260923 18:09:16.586040 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:40593
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=34261
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:16.595345 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.578981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:57517 (local address 127.18.44.254:45651)
0923 18:09:16.579184 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.579189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.580323 (+ 1134us) server_negotiation.cc:407] Connection header received
0923 18:09:16.581594 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.581601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.581689 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.581769 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.583515 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.584194 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.585881 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.586277 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.586418 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.590007 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.590012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.590015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.590060 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.591813 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.592796 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.592800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.592801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.593060 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.593648 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.593652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.593674 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.593941 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.594045 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.594896 (+ 851us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.595095 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:09:16.596529 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.579428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:57517)
0923 18:09:16.580108 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.580178 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.581294 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.582024 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.582033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.582587 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:16.583319 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.583339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.584384 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.584388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.585687 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.585702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.586553 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.586560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.586779 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.587605 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:16.587624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:16.589680 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:16.592051 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.592061 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.592084 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.592583 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.593207 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.593210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.593213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.593473 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.594075 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:16.594083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:16.594532 (+ 449us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.594780 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.595210 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260923 18:09:16.598606 2420 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:09:16.599032 2420 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:16.599850 2420 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:09:16.601593 2090 ts_manager.cc:194] Registered new tserver with Master: c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833)
I20260923 18:09:16.602490 2090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:57517
W20260923 18:09:16.714061 2423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:16.714365 2423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:16.714581 2423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:16.719246 2423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:16.719367 2423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:16.719457 2423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:09:16.725482 2423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=34261
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:16.726605 2423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:16.727728 2423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:16.740228 2429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.740228 2431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:16.740447 2428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:16.740952 2423 server_base.cc:1034] running on GCE node
I20260923 18:09:16.741564 2423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:16.742821 2423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:16.744086 2423 hybrid_clock.cc:648] HybridClock initialized: now 1790186956743994 us; error 110 us; skew 500 ppm
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:16.748040 2423 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:09:16.749502 2423 webserver.cc:460] Webserver started at http://127.18.44.195:34261/ using document root <none> and password file <none>
I20260923 18:09:16.750478 2423 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:16.750623 2423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:16.756556 2423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:16.759879 2438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.762008 2423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:09:16.762230 2423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "63ba20486c904e339ae0c9ce7511fcd3"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "8ba3a9523d4d86dd4a384ac986ddcfd7"
server_key_iv: "db97c0830ae1da784a058ff08371934c"
server_key_version: "encryptionkey@0"
I20260923 18:09:16.763065 2423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:16.785890 2423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:16.790781 2423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:16.791047 2423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:16.791910 2423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:16.793025 2423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:16.793081 2423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.793139 2423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:16.793154 2423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:16.805960 2423 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40593
I20260923 18:09:16.805979 2551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40593 every 8 connection(s)
I20260923 18:09:16.807425 2423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:09:16.815886 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2423
Sep 23 18:09:16 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186956, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:16.826313 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.810125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:42023 (local address 127.18.44.254:45651)
0923 18:09:16.810291 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.810296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.811653 (+ 1357us) server_negotiation.cc:407] Connection header received
0923 18:09:16.813212 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.813216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.813286 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.813365 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.814758 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.815352 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.816792 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.817144 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.817342 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.820629 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.820633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.820636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.820688 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.822577 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.823901 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.823905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.823907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.824067 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.824657 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.824661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.824663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.824944 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.825033 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.825962 (+ 929us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.826127 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":896}
I20260923 18:09:16.827109 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.816959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.18.44.254:45651)
0923 18:09:16.817231 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.817235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.817380 (+ 145us) server_negotiation.cc:407] Connection header received
0923 18:09:16.817546 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.817548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.817601 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.817706 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.818598 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.819239 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.820270 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.820516 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.820614 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.822729 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.822732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.822735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.822762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.824824 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.825587 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.825590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.825591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.825690 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.826063 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.826066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.826068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.826276 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.826762 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.826836 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.826951 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"spinlock_wait_cycles":2432,"thread_start_us":94,"threads_started":1}
I20260923 18:09:16.827922 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.810547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:42023)
0923 18:09:16.811365 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.811438 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.812933 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.813605 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.813614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.814034 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:16.814593 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.814609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.815507 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.815510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.816546 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.816561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.817390 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.817395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.817600 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.818253 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:16.818271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:16.820422 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:16.822801 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.822809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.822832 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.823765 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.824285 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:16.824288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:16.824290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:16.824511 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:16.825085 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:16.825096 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:16.825614 (+ 518us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.825879 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.826397 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":187,"threads_started":1}
I20260923 18:09:16.829317 2552 heartbeater.cc:347] Connected to a master server at 127.18.44.254:45651
I20260923 18:09:16.829597 2552 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:16.830191 2552 heartbeater.cc:511] Master 127.18.44.254:45651 requested a full tablet report, sending...
I20260923 18:09:16.831535 2090 ts_manager.cc:194] Registered new tserver with Master: 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:09:16.832278 2090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:42023
I20260923 18:09:16.842164 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:09:16.846890 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.842851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.18.44.254:45651)
0923 18:09:16.842994 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.842999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.843532 (+ 533us) server_negotiation.cc:407] Connection header received
0923 18:09:16.843707 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.843710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.843821 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.843917 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:09:16.844232 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.844811 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.845687 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.845908 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.845996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.846314 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:09:16.846470 (+ 156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 18:09:16.846748 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45954: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 18:09:16.846975 2556 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45954: token expired
W20260923 18:09:16.847705 2021 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.44.254:45651)
I20260923 18:09:16.857851 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.848262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.18.44.254:45651)
0923 18:09:16.848428 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.848432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.848450 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:09:16.848590 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.848593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.848646 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.848717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:16.849616 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.850210 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.851086 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.851373 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.851480 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.853255 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:16.853259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:16.853262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:16.853304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:16.855606 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.856287 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.856291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.856292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.856408 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:16.856732 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:16.856735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:16.856736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:16.856970 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:16.857091 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.857497 (+ 406us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.857650 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:09:16.861088 2090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:09:16.864826 2090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:09:16.875428 2215 tablet_service.cc:1467] Processing CreateTablet for tablet 8e54b9aeb51a47488663f803601557b3 (DEFAULT_TABLE table=test-table [id=975d4eebb94947799c2a8d98c0841b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:16.876277 2215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e54b9aeb51a47488663f803601557b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:16.879899 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.873121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46833 (local address 127.0.0.1:37172)
0923 18:09:16.873606 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.873620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.873706 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.874495 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.874498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.874513 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:16.874647 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.874654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.876240 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.876243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.877924 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.877943 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.879191 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.879195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.879378 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.879451 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.879569 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.879704 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":144}
I20260923 18:09:16.879904 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.873419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.18.44.194:46833)
0923 18:09:16.873908 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.873923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.873950 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:09:16.874046 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.874053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.874315 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.874513 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:16.874760 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.876087 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.878098 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.878992 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.879135 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.879344 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.879604 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.879698 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":202,"threads_started":1}
I20260923 18:09:16.880748 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.873309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40593 (local address 127.0.0.1:47102)
0923 18:09:16.874107 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.874124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.874237 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.874698 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.874701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.874717 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:16.874799 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.874804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.876301 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.876305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.877902 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.877914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.880308 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.880313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.880413 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.880442 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.880518 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.880609 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":150,"thread_start_us":117,"threads_started":1}
I20260923 18:09:16.881186 2564 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap starting.
I20260923 18:09:16.881331 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.873661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.18.44.195:40593)
0923 18:09:16.874183 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.874194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.874217 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:09:16.874375 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.874381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.874554 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.874701 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:16.874905 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.876087 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.878056 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.879002 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.879135 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.879344 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.880992 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.881169 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":144,"threads_started":1}
I20260923 18:09:16.882333 2564 tablet_bootstrap.cc:654] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:16.882052 2354 tablet_service.cc:1467] Processing CreateTablet for tablet 8e54b9aeb51a47488663f803601557b3 (DEFAULT_TABLE table=test-table [id=975d4eebb94947799c2a8d98c0841b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:16.883631 2354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e54b9aeb51a47488663f803601557b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:16.884241 2564 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: No bootstrap required, opened a new log
I20260923 18:09:16.884367 2564 ts_tablet_manager.cc:1397] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:09:16.884958 2564 raft_consensus.cc:348] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.885136 2564 raft_consensus.cc:374] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:16.885198 2564 raft_consensus.cc:729] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8acb5a6654709a0c5b2d69960deb0, State: Initialized, Role: FOLLOWER
I20260923 18:09:16.885406 2564 consensus_queue.cc:260] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.885183 2486 tablet_service.cc:1467] Processing CreateTablet for tablet 8e54b9aeb51a47488663f803601557b3 (DEFAULT_TABLE table=test-table [id=975d4eebb94947799c2a8d98c0841b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:16.886451 2564 ts_tablet_manager.cc:1428] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:09:16.886740 2486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e54b9aeb51a47488663f803601557b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:16.890796 2312 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Bootstrap starting.
I20260923 18:09:16.894058 2312 tablet_bootstrap.cc:654] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:16.895560 2312 log.cc:824] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:16.897990 2568 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Bootstrap starting.
I20260923 18:09:16.898550 2312 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: No bootstrap required, opened a new log
I20260923 18:09:16.898865 2312 ts_tablet_manager.cc:1397] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260923 18:09:16.900821 2568 tablet_bootstrap.cc:654] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:16.902025 2568 log.cc:824] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:16.903343 2312 raft_consensus.cc:348] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.903753 2312 raft_consensus.cc:374] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:16.903820 2312 raft_consensus.cc:729] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96d5f2301d0496fbfc771d580796061, State: Initialized, Role: FOLLOWER
I20260923 18:09:16.904390 2312 consensus_queue.cc:260] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.904808 2568 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: No bootstrap required, opened a new log
I20260923 18:09:16.905047 2568 ts_tablet_manager.cc:1397] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:09:16.905380 2420 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
I20260923 18:09:16.906188 2312 ts_tablet_manager.cc:1428] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:09:16.908634 2568 raft_consensus.cc:348] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.908999 2568 raft_consensus.cc:374] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:16.909072 2568 raft_consensus.cc:729] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ba20486c904e339ae0c9ce7511fcd3, State: Initialized, Role: FOLLOWER
I20260923 18:09:16.909564 2568 consensus_queue.cc:260] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.910617 2568 ts_tablet_manager.cc:1428] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:09:16.910889 2552 heartbeater.cc:503] Master 127.18.44.254:45651 was elected leader, sending a full tablet report...
I20260923 18:09:16.990600 2566 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:16.990980 2566 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:16.992422 2566 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:09:16.999538 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.992755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46833 (local address 127.18.44.193:48891)
0923 18:09:16.993326 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.993345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.993550 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.994030 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.994033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.994069 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:16.994243 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.994252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.995622 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.995627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.997206 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.997228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.998553 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.998558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.998882 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.999030 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.999131 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.999310 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":335,"spinlock_wait_cycles":57856,"thread_start_us":131,"threads_started":1}
I20260923 18:09:16.999575 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.992882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38945 (local address 127.18.44.195:40593)
0923 18:09:16.993027 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.993031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.993520 (+ 489us) server_negotiation.cc:407] Connection header received
0923 18:09:16.993764 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.993767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.993830 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.993937 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:16.994459 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.995407 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.997526 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.998399 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.998524 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.998596 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.999279 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.999385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:09:16.999559 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.992886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:48891 (local address 127.18.44.194:46833)
0923 18:09:16.993027 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:16.993031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:16.993598 (+ 567us) server_negotiation.cc:407] Connection header received
0923 18:09:16.993696 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:16.993700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:16.993803 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:16.993937 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:16.994462 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.995476 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.997545 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.998305 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.998524 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.998596 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:16.999226 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 18:09:16.999378 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:09:16.999538 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:16.992750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40593 (local address 127.18.44.193:38945)
0923 18:09:16.993326 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:16.993345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:16.993520 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:16.994014 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:16.994018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:16.994057 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:16.994262 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.994268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.995622 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.995627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.997206 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:16.997228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:16.998655 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:16.998660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:16.998876 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:16.999002 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 18:09:16.999126 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 18:09:16.999300 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":210,"spinlock_wait_cycles":32640,"thread_start_us":285,"threads_started":1}
I20260923 18:09:17.000922 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e54b9aeb51a47488663f803601557b3" candidate_uuid: "61f8acb5a6654709a0c5b2d69960deb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96d5f2301d0496fbfc771d580796061" is_pre_election: true
I20260923 18:09:17.000928 2505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e54b9aeb51a47488663f803601557b3" candidate_uuid: "61f8acb5a6654709a0c5b2d69960deb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ba20486c904e339ae0c9ce7511fcd3" is_pre_election: true
I20260923 18:09:17.001340 2374 raft_consensus.cc:2454] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f8acb5a6654709a0c5b2d69960deb0 in term 0.
I20260923 18:09:17.001340 2505 raft_consensus.cc:2454] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f8acb5a6654709a0c5b2d69960deb0 in term 0.
I20260923 18:09:17.002115 2168 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0, 63ba20486c904e339ae0c9ce7511fcd3; no voters:
I20260923 18:09:17.002570 2566 raft_consensus.cc:2781] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:09:17.002772 2566 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:09:17.002838 2566 raft_consensus.cc:3037] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:17.004524 2566 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:17.005085 2566 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 1 election: Requested vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
I20260923 18:09:17.005913 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e54b9aeb51a47488663f803601557b3" candidate_uuid: "61f8acb5a6654709a0c5b2d69960deb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96d5f2301d0496fbfc771d580796061"
I20260923 18:09:17.005883 2505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e54b9aeb51a47488663f803601557b3" candidate_uuid: "61f8acb5a6654709a0c5b2d69960deb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ba20486c904e339ae0c9ce7511fcd3"
I20260923 18:09:17.006095 2505 raft_consensus.cc:3037] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:17.006102 2374 raft_consensus.cc:3037] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:17.007795 2505 raft_consensus.cc:2454] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f8acb5a6654709a0c5b2d69960deb0 in term 1.
I20260923 18:09:17.007795 2374 raft_consensus.cc:2454] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f8acb5a6654709a0c5b2d69960deb0 in term 1.
I20260923 18:09:17.008364 2167 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0, c96d5f2301d0496fbfc771d580796061; no voters:
I20260923 18:09:17.008669 2566 raft_consensus.cc:2781] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:17.008985 2566 raft_consensus.cc:686] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 LEADER]: Becoming Leader. State: Replica: 61f8acb5a6654709a0c5b2d69960deb0, State: Running, Role: LEADER
I20260923 18:09:17.009370 2566 consensus_queue.cc:237] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:17.013839 2092 catalog_manager.cc:5495] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193). New cstate: current_term: 1 leader_uuid: "61f8acb5a6654709a0c5b2d69960deb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } health_report { overall_health: HEALTHY } } }
I20260923 18:09:17.056996 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:17.051403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.18.44.193:35355)
0923 18:09:17.051860 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:17.051872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:17.051890 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:09:17.052012 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:17.052015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:17.052166 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:17.052340 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:09:17.052749 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:17.053388 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:17.054431 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:17.054717 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:17.054839 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:17.055361 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:09:17.056045 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:09:17.056185 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:17.056408 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 18:09:17.056500 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
W20260923 18:09:17.058755 2553 tablet.cc:2369] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:17.062718 2374 raft_consensus.cc:1261] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Refusing update from remote peer 61f8acb5a6654709a0c5b2d69960deb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:17.062718 2505 raft_consensus.cc:1261] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Refusing update from remote peer 61f8acb5a6654709a0c5b2d69960deb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:17.063921 2574 consensus_queue.cc:1035] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:17.064323 2566 consensus_queue.cc:1035] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:17.073369 2578 mvcc.cc:205] Tried to move back new op lower bound from 7332605776119742464 to 7332605775914168320. Current Snapshot: MvccSnapshot[applied={T|T < 7332605776119742464}]
W20260923 18:09:17.078400 2421 tablet.cc:2369] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:17.079406 2579 mvcc.cc:205] Tried to move back new op lower bound from 7332605776119742464 to 7332605775914168320. Current Snapshot: MvccSnapshot[applied={T|T < 7332605776119742464}]
I20260923 18:09:17.085294 2092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:17.085594 2092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:17.089737 2092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Sending DeleteTablet for 3 replicas of tablet 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:17.090854 2354 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e54b9aeb51a47488663f803601557b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:17 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:37172
I20260923 18:09:17.091781 2486 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e54b9aeb51a47488663f803601557b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:17 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:47102
I20260923 18:09:17.092017 2588 tablet_replica.cc:330] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: stopping tablet replica
I20260923 18:09:17.092368 2589 tablet_replica.cc:330] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: stopping tablet replica
I20260923 18:09:17.092638 2588 raft_consensus.cc:2229] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:17.092890 2215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e54b9aeb51a47488663f803601557b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:17 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:49012
I20260923 18:09:17.092936 2589 raft_consensus.cc:2229] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:17.093240 2588 raft_consensus.cc:2258] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:17.093555 2589 raft_consensus.cc:2258] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:17.093967 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2131
I20260923 18:09:17.093999 2590 tablet_replica.cc:330] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: stopping tablet replica
I20260923 18:09:17.096434 2589 ts_tablet_manager.cc:1905] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:17.097451 2588 ts_tablet_manager.cc:1905] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:17.099826 2589 ts_tablet_manager.cc:1918] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:17.100066 2589 log.cc:1196] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/8e54b9aeb51a47488663f803601557b3
I20260923 18:09:17.100419 2588 ts_tablet_manager.cc:1918] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:17.100510 2588 log.cc:1196] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/8e54b9aeb51a47488663f803601557b3
I20260923 18:09:17.100641 2589 ts_tablet_manager.cc:1939] T 8e54b9aeb51a47488663f803601557b3 P 63ba20486c904e339ae0c9ce7511fcd3: Deleting consensus metadata
I20260923 18:09:17.100858 2588 ts_tablet_manager.cc:1939] T 8e54b9aeb51a47488663f803601557b3 P c96d5f2301d0496fbfc771d580796061: Deleting consensus metadata
W20260923 18:09:17.101401 2080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:35355 (error 108)
I20260923 18:09:17.102295 2078 catalog_manager.cc:4841] TS 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): tablet 8e54b9aeb51a47488663f803601557b3 (table test-table [id=975d4eebb94947799c2a8d98c0841b5c]) successfully deleted
I20260923 18:09:17.102648 2077 catalog_manager.cc:4841] TS c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): tablet 8e54b9aeb51a47488663f803601557b3 (table test-table [id=975d4eebb94947799c2a8d98c0841b5c]) successfully deleted
I20260923 18:09:17.103181 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2285
I20260923 18:09:17.103377 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:17.103008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35355 (local address 127.0.0.1:49026)
0923 18:09:17.103176 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:17.103285 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 18:09:17.104576 2080 catalog_manager.cc:4584] TS 61f8acb5a6654709a0c5b2d69960deb0 (127.18.44.193:35355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e54b9aeb51a47488663f803601557b3: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35355: connect: Connection refused (error 111)
I20260923 18:09:17.109426 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2423
I20260923 18:09:17.115159 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2060
I20260923 18:09:37.124140 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--webserver_interface=127.18.44.254
--webserver_port=43609
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:37.248281 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:37.248602 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:37.248646 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:37.252975 2592 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:09:37.253054 2592 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:09:37.253069 2592 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:09:37.253082 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:37.253103 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:37.253120 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:37.253135 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:37.253150 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:37.259048 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:37.260375 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:37.261489 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:37.272270 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:37.272418 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:37.272604 2592 server_base.cc:1034] running on GCE node
W20260923 18:09:37.272300 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:37.273380 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:37.274655 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:37.275961 2592 hybrid_clock.cc:648] HybridClock initialized: now 1790186977275895 us; error 102 us; skew 500 ppm
Sep 23 18:09:37 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186977, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:37.285800 2592 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:09:37.287104 2592 webserver.cc:460] Webserver started at http://127.18.44.254:43609/ using document root <none> and password file <none>
I20260923 18:09:37.287812 2592 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:37.287904 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:37.292886 2592 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:09:37.296231 2607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:37.298277 2592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:09:37.298511 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "554e8cf9d0584aa591707f155a898a57"
format_stamp: "Formatted at 2026-09-23 18:08:54 on dist-test-slave-rvjj"
server_key: "a8bb27bb80aca8a9da99e4979cbb44c9"
server_key_iv: "8eeb70f064d42673ff28cb814f09b890"
server_key_version: "encryptionkey@0"
I20260923 18:09:37.299180 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:37.336622 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:37.340484 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:37.340695 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:37.349071 2592 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:45651
I20260923 18:09:37.349552 2659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:45651 every 8 connection(s)
I20260923 18:09:37.350313 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:09:37.354544 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2592
I20260923 18:09:37.354990 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:35355
--local_ip_for_outbound_sockets=127.18.44.193
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=37559
--webserver_interface=127.18.44.193
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:09:37.356575 2660 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:09:37.359163 2660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap starting.
I20260923 18:09:37.382532 2660 log.cc:824] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:37.397192 2660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:09:37.397698 2660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Bootstrap complete.
I20260923 18:09:37.402721 2660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:37.403412 2660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Initialized, Role: FOLLOWER
I20260923 18:09:37.404234 2660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:37.404428 2660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:09:37.404510 2660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:09:37.404628 2660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:09:37.407492 2660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:37.407980 2660 leader_election.cc:302] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554e8cf9d0584aa591707f155a898a57; no voters:
I20260923 18:09:37.408447 2660 leader_election.cc:288] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 18:09:37.408577 2664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 18:09:37.408810 2664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [term 3 LEADER]: Becoming Leader. State: Replica: 554e8cf9d0584aa591707f155a898a57, State: Running, Role: LEADER
I20260923 18:09:37.409202 2664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } }
I20260923 18:09:37.410125 2660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:09:37.410562 2664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554e8cf9d0584aa591707f155a898a57. Latest consensus state: current_term: 3 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:09:37.410758 2664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:37.411115 2665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "554e8cf9d0584aa591707f155a898a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8cf9d0584aa591707f155a898a57" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 45651 } } }
I20260923 18:09:37.411440 2665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:37.412984 2668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:09:37.419037 2668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:09:37.419960 2668 catalog_manager.cc:1259] Loaded cluster ID: cca55a528c9a46c3853400200eb2eac4
I20260923 18:09:37.420040 2668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:09:37.422704 2668 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:09:37.423750 2668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Loaded TSK: 0
I20260923 18:09:37.423870 2668 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 554e8cf9d0584aa591707f155a898a57: Loaded TSK: 1
F20260923 18:09:37.424687 2668 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbe48ce29ad google::LogMessage::Fail() at ??:0
@ 0x7fbe48ce588a google::LogMessage::SendToLog() at ??:0
@ 0x7fbe48ce24de google::LogMessage::Flush() at ??:0
@ 0x7fbe48ce3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe4b9a618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbe4b99ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbe5364b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbe5362599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbe53626178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbe53667df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbe5368682b std::function<>::operator()() at ??:0
@ 0x7fbe53625b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbe53626f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe53623579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbe53667640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe4af343d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe4af34c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbe4af363f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe4af25512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe4c5696db start_thread at ??:0
@ 0x7fbe477e061f clone at ??:0
W20260923 18:09:37.506448 2662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:37.506898 2662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:37.506978 2662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:37.513046 2662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:37.513144 2662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:37.513221 2662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:09:37.520874 2662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:35355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=37559
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:37.522634 2662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:37.524245 2662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:37.538142 2710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:37.538172 2707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:37.538414 2662 server_base.cc:1034] running on GCE node
W20260923 18:09:37.538542 2708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:37.539309 2662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:37.540313 2662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:37.541656 2662 hybrid_clock.cc:648] HybridClock initialized: now 1790186977541604 us; error 64 us; skew 500 ppm
Sep 23 18:09:37 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186977, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:37.545881 2662 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:09:37.547573 2662 webserver.cc:460] Webserver started at http://127.18.44.193:37559/ using document root <none> and password file <none>
I20260923 18:09:37.548413 2662 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:37.548514 2662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:37.554643 2662 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:09:37.558075 2717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:37.560623 2662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:09:37.560844 2662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "61f8acb5a6654709a0c5b2d69960deb0"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "b9b2560afc2c1b6c07a6460902510335"
server_key_iv: "ac38ab9294c21e69e88bb9f7120fd411"
server_key_version: "encryptionkey@0"
I20260923 18:09:37.561682 2662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:37.576508 2662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:37.580888 2662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:37.581140 2662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:37.581929 2662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:37.583400 2724 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:09:37.588754 2662 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:09:37.588860 2662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260923 18:09:37.588927 2662 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:09:37.591316 2662 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:09:37.591415 2662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260923 18:09:37.591758 2724 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap starting.
I20260923 18:09:37.604338 2662 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:35355
I20260923 18:09:37.605700 2831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:35355 every 8 connection(s)
I20260923 18:09:37.606352 2662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:09:37.613557 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2662
I20260923 18:09:37.614148 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:46833
--local_ip_for_outbound_sockets=127.18.44.194
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=38313
--webserver_interface=127.18.44.194
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:09:37.619570 2724 log.cc:824] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:37.633124 2724 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:09:37.633725 2724 tablet_bootstrap.cc:492] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Bootstrap complete.
I20260923 18:09:37.635370 2724 ts_tablet_manager.cc:1397] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.013s
I20260923 18:09:37.639680 2724 raft_consensus.cc:348] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:37.640452 2724 raft_consensus.cc:729] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f8acb5a6654709a0c5b2d69960deb0, State: Initialized, Role: FOLLOWER
I20260923 18:09:37.641158 2724 consensus_queue.cc:260] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:37.642655 2724 ts_tablet_manager.cc:1428] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260923 18:09:37.666626 2834 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:09:37.609883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:41587)
0923 18:09:37.610765 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.610843 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:37.612094 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:37.665526 (+ 53432us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":286,"threads_started":1}
W20260923 18:09:37.667171 2832 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45651 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45651: Client connection negotiation failed: client connection to 127.18.44.254:45651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:09:37.668006 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:37.667729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:52453)
0923 18:09:37.667863 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.667941 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 18:09:37.668612 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:37.668417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:33659)
0923 18:09:37.668520 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.668563 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 18:09:37.668761 2832 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:09:37.774412 2835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:37.774848 2835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:37.774921 2835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:37.779795 2835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:37.779920 2835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:37.780027 2835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:09:37.787063 2835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:46833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=38313
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:37.788688 2835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:37.790164 2835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:37.802958 2845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:37.802958 2843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:37.802999 2842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:37.803565 2835 server_base.cc:1034] running on GCE node
I20260923 18:09:37.804131 2835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:37.805012 2835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:37.806243 2835 hybrid_clock.cc:648] HybridClock initialized: now 1790186977806256 us; error 81 us; skew 500 ppm
Sep 23 18:09:37 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186977, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:37.810003 2835 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:09:37.811347 2835 webserver.cc:460] Webserver started at http://127.18.44.194:38313/ using document root <none> and password file <none>
I20260923 18:09:37.812098 2835 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:37.812285 2835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:37.817729 2835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:09:37.820897 2852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:37.823096 2835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:09:37.823581 2835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "c96d5f2301d0496fbfc771d580796061"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "454cd99afeaa0d99e1c9f97277da4d35"
server_key_iv: "91f68cf3f59df5d4e56034fd52bb3238"
server_key_version: "encryptionkey@0"
I20260923 18:09:37.824700 2835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:37.867341 2835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:37.871671 2835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:37.871891 2835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:37.872691 2835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:37.873875 2835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:37.873929 2835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:37.873975 2835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:37.873998 2835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:37.885640 2835 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:46833
I20260923 18:09:37.885725 2965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:46833 every 8 connection(s)
I20260923 18:09:37.887001 2835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:09:37.892174 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:37.889897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:43605)
0923 18:09:37.890552 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.890708 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":155,"threads_started":1}
W20260923 18:09:37.892930 2966 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45651 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45651: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
I20260923 18:09:37.893836 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:37.893589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:33955)
0923 18:09:37.893722 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.893776 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 18:09:37.894524 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:37.894329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:59343)
0923 18:09:37.894429 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:37.894474 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 18:09:37.894683 2966 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:09:37.895839 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2835
I20260923 18:09:37.896207 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:40593
--local_ip_for_outbound_sockets=127.18.44.195
--tserver_master_addrs=127.18.44.254:45651
--webserver_port=34261
--webserver_interface=127.18.44.195
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:38.038587 2969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:38.039021 2969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:38.039089 2969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:38.043957 2969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:38.044081 2969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:38.044189 2969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:09:38.052654 2969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=34261
--tserver_master_addrs=127.18.44.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:38.054148 2969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:38.055464 2969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:38.067412 2977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:38.067476 2975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:38.067636 2969 server_base.cc:1034] running on GCE node
W20260923 18:09:38.067425 2974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:38.068478 2969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:38.069469 2969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:38.070713 2969 hybrid_clock.cc:648] HybridClock initialized: now 1790186978070689 us; error 43 us; skew 500 ppm
Sep 23 18:09:38 dist-test-slave-rvjj krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186978, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:38.074823 2969 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:09:38.076274 2969 webserver.cc:460] Webserver started at http://127.18.44.195:34261/ using document root <none> and password file <none>
I20260923 18:09:38.077070 2969 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:38.077179 2969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:38.082739 2969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:09:38.086124 2984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:38.088119 2969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:38.088318 2969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "63ba20486c904e339ae0c9ce7511fcd3"
format_stamp: "Formatted at 2026-09-23 18:08:55 on dist-test-slave-rvjj"
server_key: "8ba3a9523d4d86dd4a384ac986ddcfd7"
server_key_iv: "db97c0830ae1da784a058ff08371934c"
server_key_version: "encryptionkey@0"
I20260923 18:09:38.089151 2969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:38.106699 2969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:38.111475 2969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:38.111711 2969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:38.112490 2969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:38.113698 2969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:38.113770 2969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:38.113837 2969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:38.113862 2969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:38.126435 2969 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40593
I20260923 18:09:38.126493 3097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40593 every 8 connection(s)
I20260923 18:09:38.128388 2969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:09:38.134754 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.131852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:57875)
0923 18:09:38.132824 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.133034 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":228,"threads_started":1}
W20260923 18:09:38.135787 3098 heartbeater.cc:644] Failed to heartbeat to 127.18.44.254:45651 (0 consecutive failures): Network error: Failed to ping master at 127.18.44.254:45651: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
I20260923 18:09:38.136165 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 2969
I20260923 18:09:38.136687 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.136423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:46457)
0923 18:09:38.136546 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.136608 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 18:09:38.137603 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.137344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:57851)
0923 18:09:38.137482 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.137534 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 18:09:38.137856 3098 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:09:38.670300 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.669605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:48463)
0923 18:09:38.670002 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.670153 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":90,"threads_started":1}
I20260923 18:09:38.896234 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.895578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:55993)
0923 18:09:38.895981 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.896072 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260923 18:09:38.929566 3104 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:38.929812 3104 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:38.931146 3104 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
Sep 23 18:09:38 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186977, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.194@KRBTEST.COM
Sep 23 18:09:38 dist-test-slave-rvjj krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186977, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.195@KRBTEST.COM
I20260923 18:09:38.948900 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.931798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:49007 (local address 127.18.44.194:46833)
0923 18:09:38.932250 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:38.932274 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:38.932317 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:09:38.933163 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:38.933171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:38.933593 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:38.933914 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:38.935824 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.937618 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.939343 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.939961 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.940180 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:38.943554 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:38.943561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:38.943575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:38.943617 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:38.945696 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946371 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:38.946378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:38.946381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:38.946516 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946962 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:38.946966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:38.946967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:38.947575 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:38.947722 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:38.948362 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 18:09:38.948694 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":87,"threads_started":1}
I20260923 18:09:38.948909 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.931454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40593 (local address 127.18.44.193:57577)
0923 18:09:38.931702 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.931722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:38.931898 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:38.934157 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:38.934161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:38.934828 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:38.935515 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:38.935540 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.937867 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.937871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:38.939145 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:38.939159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.940140 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.940144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:38.940389 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:38.941194 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:38.941222 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:38.943564 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:38.945997 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:38.946003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:38.946280 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:38.946760 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:38.946766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:38.946869 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:38.947832 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:38.947835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:38.948009 (+ 174us) client_negotiation.cc:769] Sending connection context
0923 18:09:38.948211 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 18:09:38.948669 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":191,"spinlock_wait_cycles":30720}
I20260923 18:09:38.948909 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.931709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:57577 (local address 127.18.44.195:40593)
0923 18:09:38.932415 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:38.932436 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:38.932485 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:09:38.933316 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:38.933323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:38.933781 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:38.934162 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:38.935796 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.937617 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.939407 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.939983 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.940166 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:38.943913 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:38.943920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:38.943941 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:38.943991 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:38.945842 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946426 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:38.946433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:38.946440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:38.946638 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946992 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:38.946995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:38.946997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:38.947718 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:38.947858 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:38.948353 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 18:09:38.948676 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":175,"threads_started":1}
I20260923 18:09:38.948987 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:38.931729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:46833 (local address 127.18.44.193:49007)
0923 18:09:38.932197 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:38.932212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:38.932321 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:38.934093 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:38.934102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:38.934839 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:38.935516 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:38.935540 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.937871 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.937874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:38.939153 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:38.939168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:38.940142 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:38.940154 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:38.940388 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:38.941201 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:38.941217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:38.943195 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:38.945857 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:38.945865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:38.945881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:38.946235 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:38.946653 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:38.946656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:38.946658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:38.946848 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:38.947707 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:38.947714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:38.948011 (+ 297us) client_negotiation.cc:769] Sending connection context
0923 18:09:38.948205 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 18:09:38.948770 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":282,"spinlock_wait_cycles":28800,"thread_start_us":132,"threads_started":1}
W20260923 18:09:38.952636 2718 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
W20260923 18:09:38.952900 2719 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:38.953078 2719 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0; no voters: 63ba20486c904e339ae0c9ce7511fcd3, c96d5f2301d0496fbfc771d580796061
I20260923 18:09:38.953441 3104 raft_consensus.cc:2726] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:09:39.139552 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:39.138771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:56267)
0923 18:09:39.139175 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:39.139384 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":95,"threads_started":1}
I20260923 18:09:39.672060 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:39.671366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:49149)
0923 18:09:39.671795 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:39.671895 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":83,"threads_started":1}
I20260923 18:09:39.898020 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:39.897432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:56005)
0923 18:09:39.897819 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:39.897905 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260923 18:09:40.141459 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:40.140735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:46043)
0923 18:09:40.141131 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:40.141292 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260923 18:09:40.621740 3112 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:40.621961 3112 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:40.622394 3112 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
W20260923 18:09:40.623690 2719 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
W20260923 18:09:40.623838 2718 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:40.623950 2718 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0; no voters: 63ba20486c904e339ae0c9ce7511fcd3, c96d5f2301d0496fbfc771d580796061
I20260923 18:09:40.624109 3112 raft_consensus.cc:2726] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:09:40.673746 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:40.673166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:34645)
0923 18:09:40.673550 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:40.673635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260923 18:09:40.899716 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:40.899069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:47279)
0923 18:09:40.899469 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:40.899551 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":101,"threads_started":1}
I20260923 18:09:41.143337 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:41.142664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:40709)
0923 18:09:41.143088 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:41.143181 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":114,"threads_started":1}
I20260923 18:09:41.675491 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:41.674837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:38015)
0923 18:09:41.675215 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:41.675331 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260923 18:09:41.901544 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:41.900844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:53051)
0923 18:09:41.901264 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:41.901371 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
I20260923 18:09:42.145098 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:42.144369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:34291)
0923 18:09:42.144851 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:42.144952 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":83,"threads_started":1}
I20260923 18:09:42.191696 3119 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:42.191910 3119 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:42.192559 3119 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
W20260923 18:09:42.193660 2718 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
W20260923 18:09:42.193850 2719 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:42.193917 2719 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0; no voters: 63ba20486c904e339ae0c9ce7511fcd3, c96d5f2301d0496fbfc771d580796061
I20260923 18:09:42.194154 3119 raft_consensus.cc:2726] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:09:42.677527 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:42.676947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:53641)
0923 18:09:42.677326 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:42.677401 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
I20260923 18:09:42.903494 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:42.902751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:43161)
0923 18:09:42.903231 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:42.903361 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260923 18:09:43.150768 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:43.146161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:57137)
0923 18:09:43.150471 (+ 4310us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:43.150596 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4153,"thread_start_us":86,"threads_started":1}
I20260923 18:09:43.679600 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:43.678797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:53693)
0923 18:09:43.679347 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:43.679454 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":141,"threads_started":1}
I20260923 18:09:43.905241 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:43.904614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:46999)
0923 18:09:43.904994 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:43.905094 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
I20260923 18:09:44.153012 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:44.152359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:58939)
0923 18:09:44.152787 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:44.152879 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":92,"threads_started":1}
I20260923 18:09:44.681704 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:44.680974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:51551)
0923 18:09:44.681491 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:44.681587 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":134,"threads_started":1}
I20260923 18:09:44.907215 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:44.906419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:43353)
0923 18:09:44.907013 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:44.907098 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":178,"threads_started":1}
I20260923 18:09:45.154949 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:45.154157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:51083)
0923 18:09:45.154668 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:45.154752 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":151,"threads_started":1}
I20260923 18:09:45.683390 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:45.682891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:54005)
0923 18:09:45.683185 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:45.683238 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":127,"threads_started":1}
I20260923 18:09:45.909264 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:45.908514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:56035)
0923 18:09:45.908956 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:45.909096 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260923 18:09:46.156755 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:46.156117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:44427)
0923 18:09:46.156514 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:46.156611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260923 18:09:46.685017 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:46.684492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:44455)
0923 18:09:46.684840 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:46.684906 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":215,"threads_started":1}
I20260923 18:09:46.880405 3133 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:46.880571 3133 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:46.881299 3133 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
W20260923 18:09:46.882409 2718 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
W20260923 18:09:46.882642 2719 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:46.882754 2719 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0; no voters: 63ba20486c904e339ae0c9ce7511fcd3, c96d5f2301d0496fbfc771d580796061
I20260923 18:09:46.882998 3133 raft_consensus.cc:2726] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:09:46.910912 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:46.910390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:40587)
0923 18:09:46.910693 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:46.910749 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260923 18:09:47.158739 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:47.157963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:39217)
0923 18:09:47.158514 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:47.158615 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":200,"threads_started":1}
I20260923 18:09:47.687038 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:47.686413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:36973)
0923 18:09:47.686799 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:47.686882 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260923 18:09:47.912732 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:47.912092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:46443)
0923 18:09:47.912528 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:47.912618 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260923 18:09:48.160681 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:48.160024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:39115)
0923 18:09:48.160469 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:48.160564 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":163,"threads_started":1}
I20260923 18:09:48.688992 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:48.688128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:50489)
0923 18:09:48.688680 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:48.688866 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":124,"threads_started":1}
I20260923 18:09:48.914448 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:48.913770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:33607)
0923 18:09:48.914177 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:48.914333 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260923 18:09:49.162441 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:49.161732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:41975)
0923 18:09:49.162212 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:49.162290 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":138,"threads_started":1}
I20260923 18:09:49.690850 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:49.690194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:33263)
0923 18:09:49.690612 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:49.690710 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
I20260923 18:09:49.916185 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:49.915608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:40363)
0923 18:09:49.915994 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:49.916076 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260923 18:09:50.164175 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:50.163465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:54531)
0923 18:09:50.163856 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:50.163956 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
I20260923 18:09:50.372020 3145 raft_consensus.cc:482] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:50.372186 3145 raft_consensus.cc:504] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96d5f2301d0496fbfc771d580796061" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 46833 } } peers { permanent_uuid: "63ba20486c904e339ae0c9ce7511fcd3" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40593 } } peers { permanent_uuid: "61f8acb5a6654709a0c5b2d69960deb0" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35355 } }
I20260923 18:09:50.372725 3145 leader_election.cc:288] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833), 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593)
W20260923 18:09:50.373762 2718 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c96d5f2301d0496fbfc771d580796061 (127.18.44.194:46833): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
W20260923 18:09:50.373937 2719 leader_election.cc:341] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63ba20486c904e339ae0c9ce7511fcd3 (127.18.44.195:40593): Not found: Tablet not found: 8e54b9aeb51a47488663f803601557b3
I20260923 18:09:50.374056 2719 leader_election.cc:302] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61f8acb5a6654709a0c5b2d69960deb0; no voters: 63ba20486c904e339ae0c9ce7511fcd3, c96d5f2301d0496fbfc771d580796061
I20260923 18:09:50.374248 3145 raft_consensus.cc:2726] T 8e54b9aeb51a47488663f803601557b3 P 61f8acb5a6654709a0c5b2d69960deb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:09:50.692688 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:50.692066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:58975)
0923 18:09:50.692429 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:50.692516 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":89,"threads_started":1}
I20260923 18:09:50.917887 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:50.917304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:42833)
0923 18:09:50.917684 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:50.917758 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260923 18:09:51.165957 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:51.165207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:34265)
0923 18:09:51.165698 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:51.165820 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":121,"threads_started":1}
I20260923 18:09:51.694259 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:51.693715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:44651)
0923 18:09:51.694075 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:51.694154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260923 18:09:51.919735 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:51.919042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:54971)
0923 18:09:51.919523 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:51.919610 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":147,"threads_started":1}
I20260923 18:09:52.167687 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:52.166992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.195:42965)
0923 18:09:52.167456 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:52.167552 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260923 18:09:52.696573 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:52.695890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.193:58221)
0923 18:09:52.696328 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:52.696426 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":88,"threads_started":1}
I20260923 18:09:52.921518 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:52.920773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:45651 (local address 127.18.44.194:40047)
0923 18:09:52.921280 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:52.921381 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.254:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260923 18:09:53.145012 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2662
I20260923 18:09:53.155422 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2835
I20260923 18:09:53.164784 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 2969
2026-09-23T18:09:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59780 ms)
[----------] 30 tests from SecurityITest (266039 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:09:53 dist-test-slave-rvjj krb5kdc[3156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:09:53 dist-test-slave-rvjj krb5kdc[3156](info): set up 2 sockets
Sep 23 18:09:53 dist-test-slave-rvjj krb5kdc[3156](info): commencing operation
krb5kdc: starting...
W20260923 18:09:54.160235 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:09:54 dist-test-slave-rvjj krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:09:54.292872 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33297
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:33297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:54.418889 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:54.419220 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:54.419319 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:54.423863 3169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:54.423983 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:54.424015 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:54.424037 3169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:54.424057 3169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:54.430115 3169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:33297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:33297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:54.431499 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:54.432956 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:54.444959 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:54.445171 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:54.445089 3169 server_base.cc:1034] running on GCE node
W20260923 18:09:54.445602 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:54.446410 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:54.447749 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:54.448946 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790186994448901 us; error 67 us; skew 500 ppm
Sep 23 18:09:54 dist-test-slave-rvjj krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:54.452745 3169 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:09:54.453894 3169 webserver.cc:460] Webserver started at http://127.18.44.254:35833/ using document root <none> and password file <none>
I20260923 18:09:54.454576 3169 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:54.454654 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:54.454881 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:54.457281 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "878323d29b4a404c81ab698ae1c3d39d"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "6ff7bc0e8daacd153f190f9f291af1e4"
server_key_iv: "bc042e0a2ea5dfb880aec8ed32eb341e"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.457924 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "878323d29b4a404c81ab698ae1c3d39d"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "6ff7bc0e8daacd153f190f9f291af1e4"
server_key_iv: "bc042e0a2ea5dfb880aec8ed32eb341e"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.463675 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:09:54.467468 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:54.469476 3169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:54.469693 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "878323d29b4a404c81ab698ae1c3d39d"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "6ff7bc0e8daacd153f190f9f291af1e4"
server_key_iv: "bc042e0a2ea5dfb880aec8ed32eb341e"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.469847 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:54.484565 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:54.489425 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:54.489666 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:54.499930 3169 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:33297
I20260923 18:09:54.500037 3236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:33297 every 8 connection(s)
I20260923 18:09:54.501176 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:09:54.506418 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:54.511145 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3169
I20260923 18:09:54.511334 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:09:54.511762 18611 external_mini_cluster.cc:1428] Setting key 45dd9624a780e73f153325b50330dbce
I20260923 18:09:54.515475 3237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: Bootstrap starting.
I20260923 18:09:54.519655 3237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:54.522181 3237 log.cc:824] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:09:54 dist-test-slave-rvjj krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:54.527685 3237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: No bootstrap required, opened a new log
I20260923 18:09:54.533344 3237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } }
I20260923 18:09:54.533784 3237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:54.533862 3237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878323d29b4a404c81ab698ae1c3d39d, State: Initialized, Role: FOLLOWER
I20260923 18:09:54.534826 3237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } }
I20260923 18:09:54.535063 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:54.513886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.18.44.254:33297)
0923 18:09:54.514442 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:54.514461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:54.514503 (+ 42us) server_negotiation.cc:407] Connection header received
0923 18:09:54.515433 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:54.515459 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:54.515916 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:54.516268 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:54.517946 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.519139 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.521098 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.522085 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.522247 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:54.528072 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:54.528081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:54.528105 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:54.528160 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:54.530248 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:54.531139 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:54.531146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:54.531161 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:54.531416 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:54.531902 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:54.531905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:54.531906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:54.532466 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:54.532685 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:54.533231 (+ 546us) server_negotiation.cc:299] Negotiation successful
0923 18:09:54.533666 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":175,"threads_started":1}
I20260923 18:09:54.535495 3237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:09:54.535602 3237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:09:54.535820 3237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:54.538949 3237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } }
I20260923 18:09:54.539659 3237 leader_election.cc:302] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878323d29b4a404c81ab698ae1c3d39d; no voters:
I20260923 18:09:54.540232 3237 leader_election.cc:288] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:09:54.540588 3242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:54.541127 3242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [term 1 LEADER]: Becoming Leader. State: Replica: 878323d29b4a404c81ab698ae1c3d39d, State: Running, Role: LEADER
I20260923 18:09:54.541898 3242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } }
I20260923 18:09:54.542177 3237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:09:54.544174 3243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878323d29b4a404c81ab698ae1c3d39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } } }
I20260923 18:09:54.544442 3243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:54.545291 3244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878323d29b4a404c81ab698ae1c3d39d. Latest consensus state: current_term: 1 leader_uuid: "878323d29b4a404c81ab698ae1c3d39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878323d29b4a404c81ab698ae1c3d39d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 33297 } } }
I20260923 18:09:54.545785 3244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:54.545917 3253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:09:54.555694 3253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:09:54.570732 3253 catalog_manager.cc:1347] Generated new cluster ID: 76251f9d449f46499aef74954e90b3ee
I20260923 18:09:54.570888 3253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:09:54.586019 3253 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:09:54.587805 3253 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:09:54.599488 3253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: Generated new TSK 0
I20260923 18:09:54.600872 3253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:09:54.658944 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:54.826002 3263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:54.826453 3263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:54.826509 3263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:54.833069 3263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:54.833261 3263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:54.833341 3263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:09:54.842540 3263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:54.844362 3263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:54.846248 3263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:54.861178 3268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:54.861423 3263 server_base.cc:1034] running on GCE node
W20260923 18:09:54.861212 3271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:54.861212 3269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:54.862481 3263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:54.863531 3263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:54.864791 3263 hybrid_clock.cc:648] HybridClock initialized: now 1790186994864752 us; error 63 us; skew 500 ppm
Sep 23 18:09:54 dist-test-slave-rvjj krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:54.869189 3263 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:09:54.870718 3263 webserver.cc:460] Webserver started at http://127.18.44.193:34843/ using document root <none> and password file <none>
I20260923 18:09:54.871573 3263 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:54.871675 3263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:54.871929 3263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:54.874843 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "9dc633c82c314b129bcff9ebf98b641e"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "709efb68373e6e9b37ac79ffa87cf6b9"
server_key_iv: "41d0555f2fccd4ec3040610cc22b36be"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.875674 3263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "9dc633c82c314b129bcff9ebf98b641e"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "709efb68373e6e9b37ac79ffa87cf6b9"
server_key_iv: "41d0555f2fccd4ec3040610cc22b36be"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.881753 3263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 18:09:54.886564 3278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:54.889176 3263 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 18:09:54.889415 3263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "9dc633c82c314b129bcff9ebf98b641e"
format_stamp: "Formatted at 2026-09-23 18:09:54 on dist-test-slave-rvjj"
server_key: "709efb68373e6e9b37ac79ffa87cf6b9"
server_key_iv: "41d0555f2fccd4ec3040610cc22b36be"
server_key_version: "encryptionkey@0"
I20260923 18:09:54.889587 3263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:54.930056 3263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:54.935070 3263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:54.935421 3263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:54.936338 3263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:54.937539 3263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:54.937598 3263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:54.937662 3263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:54.937692 3263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:54.950785 3263 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:36459
I20260923 18:09:54.950821 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:36459 every 8 connection(s)
I20260923 18:09:54.952131 3263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:09:54.961583 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3263
I20260923 18:09:54.961745 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:09:54.962144 18611 external_mini_cluster.cc:1428] Setting key 5ab4d1421d1444b11d8653d58256dc93
Sep 23 18:09:54 dist-test-slave-rvjj krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:54.970772 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:54.954772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:40983 (local address 127.18.44.254:33297)
0923 18:09:54.955034 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:54.955038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:54.956359 (+ 1321us) server_negotiation.cc:407] Connection header received
0923 18:09:54.957985 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:54.957989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:54.958053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:54.958133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:54.959603 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.960163 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.961485 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.961744 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.961850 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:54.965408 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:54.965413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:54.965416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:54.965462 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:54.967505 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:54.968347 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:54.968352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:54.968354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:54.968486 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:54.969012 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:54.969015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:54.969016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:54.969327 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:54.969453 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:54.970293 (+ 840us) server_negotiation.cc:299] Negotiation successful
0923 18:09:54.970527 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260923 18:09:54.971660 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:54.955319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33297 (local address 127.18.44.193:40983)
0923 18:09:54.956081 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:54.956192 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:54.957713 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:54.958359 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:54.958367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:54.958789 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:54.959427 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:54.959445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.960299 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.960302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:54.961327 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:54.961339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:54.961919 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:54.961923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:54.962110 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:54.962763 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:54.962782 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:54.965076 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:54.967743 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:54.967751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:54.967770 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:54.968186 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:54.968647 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:54.968650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:54.968652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:54.968878 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:54.969502 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:54.969508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:54.969943 (+ 435us) client_negotiation.cc:769] Sending connection context
0923 18:09:54.970152 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 18:09:54.970520 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":189,"threads_started":1}
I20260923 18:09:54.973379 3392 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33297
I20260923 18:09:54.973668 3392 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:54.974262 3392 heartbeater.cc:511] Master 127.18.44.254:33297 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:09:54.976564 3201 ts_manager.cc:194] Registered new tserver with Master: 9dc633c82c314b129bcff9ebf98b641e (127.18.44.193:36459)
I20260923 18:09:54.978111 3201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:40983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:09:55.005249 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:55.170599 3397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:55.171022 3397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:55.171075 3397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:55.175866 3397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:55.175982 3397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:55.176061 3397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:09:55.182721 3397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:55.184291 3397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:55.185633 3397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:55.198685 3403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:55.198734 3402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:55.198802 3397 server_base.cc:1034] running on GCE node
W20260923 18:09:55.198725 3405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:55.199791 3397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:55.200614 3397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:55.201953 3397 hybrid_clock.cc:648] HybridClock initialized: now 1790186995201927 us; error 62 us; skew 500 ppm
Sep 23 18:09:55 dist-test-slave-rvjj krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:55.205492 3397 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:09:55.206609 3397 webserver.cc:460] Webserver started at http://127.18.44.194:41649/ using document root <none> and password file <none>
I20260923 18:09:55.207312 3397 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:55.207412 3397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:55.207664 3397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:55.209957 3397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "027777d4c108455c8baf87a126b4e7b9"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "76904684420b255160828bad213b8cbd"
server_key_iv: "3ab808ff2632d2ab3436133c0176ea8c"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.210561 3397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "027777d4c108455c8baf87a126b4e7b9"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "76904684420b255160828bad213b8cbd"
server_key_iv: "3ab808ff2632d2ab3436133c0176ea8c"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.215232 3397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:09:55.218505 3412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.220530 3397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260923 18:09:55.220744 3397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "027777d4c108455c8baf87a126b4e7b9"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "76904684420b255160828bad213b8cbd"
server_key_iv: "3ab808ff2632d2ab3436133c0176ea8c"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.220881 3397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:55.236860 3397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:55.244784 3397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:55.245013 3397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:55.245759 3397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:55.246797 3397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:55.246858 3397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.246922 3397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:55.246958 3397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.258404 3397 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:41643
I20260923 18:09:55.258460 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:41643 every 8 connection(s)
I20260923 18:09:55.259517 3397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:09:55.266157 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3397
I20260923 18:09:55.266300 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:09:55.266630 18611 external_mini_cluster.cc:1428] Setting key 5cba6cae68210f7b4aa8a1870b11a697
Sep 23 18:09:55 dist-test-slave-rvjj krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
I20260923 18:09:55.280539 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.262453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:36207 (local address 127.18.44.254:33297)
0923 18:09:55.262648 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.262652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.263868 (+ 1216us) server_negotiation.cc:407] Connection header received
0923 18:09:55.265672 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.265677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.265761 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.265860 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:55.267777 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.268567 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.270897 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.271188 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.271402 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.274929 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:55.274935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:55.274976 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:55.275023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:55.277074 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.278143 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.278147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.278149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.278275 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.278806 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.278810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.278812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.279096 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:55.279220 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.280221 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.280376 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 18:09:55.282001 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.262870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33297 (local address 127.18.44.194:36207)
0923 18:09:55.263577 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.263649 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.265308 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.266259 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.266270 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.266787 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:55.267551 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.267576 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.268770 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.268775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.270526 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.270553 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.271497 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.271503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.271736 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.272411 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:55.272431 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:55.274584 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:55.277355 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:55.277365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:55.277391 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:55.277952 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:55.278419 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:55.278422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:55.278424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:55.278677 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:55.279340 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:55.279350 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:55.279714 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.280155 (+ 441us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.280776 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:09:55.284221 3526 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33297
I20260923 18:09:55.284650 3526 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:55.285475 3526 heartbeater.cc:511] Master 127.18.44.254:33297 requested a full tablet report, sending...
I20260923 18:09:55.287127 3201 ts_manager.cc:194] Registered new tserver with Master: 027777d4c108455c8baf87a126b4e7b9 (127.18.44.194:41643)
I20260923 18:09:55.287941 3201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:36207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:09:55.307873 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:55.444942 3531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:55.445276 3531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:55.445324 3531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:55.449705 3531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:55.449801 3531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:55.449865 3531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:09:55.455909 3531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:55.457130 3531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:55.458276 3531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:55.469115 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:55.469173 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:55.469101 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:55.469411 3531 server_base.cc:1034] running on GCE node
I20260923 18:09:55.470266 3531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:55.471693 3531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:55.472947 3531 hybrid_clock.cc:648] HybridClock initialized: now 1790186995472904 us; error 66 us; skew 500 ppm
Sep 23 18:09:55 dist-test-slave-rvjj krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:55.477695 3531 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:09:55.479357 3531 webserver.cc:460] Webserver started at http://127.18.44.195:34889/ using document root <none> and password file <none>
I20260923 18:09:55.480304 3531 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:55.480460 3531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:55.480931 3531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:55.484367 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "76e8facab36d4186aa9c51774163c1eb"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "b7f76aa448e73b9e2ed854014e6e56ba"
server_key_iv: "06d791d9f950bd7e1a28d9f7e25fc396"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.485214 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "76e8facab36d4186aa9c51774163c1eb"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "b7f76aa448e73b9e2ed854014e6e56ba"
server_key_iv: "06d791d9f950bd7e1a28d9f7e25fc396"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.490901 3531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 18:09:55.495056 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.497478 3531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:55.497684 3531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "76e8facab36d4186aa9c51774163c1eb"
format_stamp: "Formatted at 2026-09-23 18:09:55 on dist-test-slave-rvjj"
server_key: "b7f76aa448e73b9e2ed854014e6e56ba"
server_key_iv: "06d791d9f950bd7e1a28d9f7e25fc396"
server_key_version: "encryptionkey@0"
I20260923 18:09:55.497850 3531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:55.536834 3531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:55.541519 3531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:55.541769 3531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:55.542616 3531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:55.543859 3531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:55.543927 3531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.543993 3531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:55.544023 3531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:55.556900 3531 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40899
I20260923 18:09:55.556921 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40899 every 8 connection(s)
I20260923 18:09:55.558265 3531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:09:55.558729 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3531
I20260923 18:09:55.558871 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:09:55.559216 18611 external_mini_cluster.cc:1428] Setting key 9ddd408e62cd11b404f27e2b64447c90
Sep 23 18:09:55 dist-test-slave-rvjj krb5kdc[3156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:55.579752 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.561153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50543 (local address 127.18.44.254:33297)
0923 18:09:55.561429 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.561434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.563079 (+ 1645us) server_negotiation.cc:407] Connection header received
0923 18:09:55.564895 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.564899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.564964 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.565083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:55.567039 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.567761 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.569454 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.569694 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.569806 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.574248 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:55.574253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:55.574256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:55.574301 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:55.576419 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.577261 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.577265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.577267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.577389 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.577981 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.577995 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.577997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.578397 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:55.578526 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.579241 (+ 715us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.579571 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260923 18:09:55.580639 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.561685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:33297 (local address 127.18.44.195:50543)
0923 18:09:55.562826 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.562925 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.564596 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.565474 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.565485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.566019 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:55.566775 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.566800 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.567967 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.567971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.569263 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.569282 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.569941 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.569944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.570239 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.571010 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:55.571033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:55.573939 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:55.576630 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:55.576638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:55.576662 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:55.577070 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:55.577519 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:55.577522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:55.577523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:55.577787 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:55.578565 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:55.578572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:55.578915 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.579128 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.579645 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":207,"threads_started":1}
I20260923 18:09:55.582568 3660 heartbeater.cc:347] Connected to a master server at 127.18.44.254:33297
I20260923 18:09:55.582942 3660 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:55.583650 3660 heartbeater.cc:511] Master 127.18.44.254:33297 requested a full tablet report, sending...
I20260923 18:09:55.585274 3201 ts_manager.cc:194] Registered new tserver with Master: 76e8facab36d4186aa9c51774163c1eb (127.18.44.195:40899)
I20260923 18:09:55.586122 3201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:50543
I20260923 18:09:55.588742 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:09:55.600742 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.590993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.18.44.254:33297)
0923 18:09:55.591193 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.591197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.591216 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:09:55.591362 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.591365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.591421 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.591520 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:55.592339 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.593205 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.594403 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.594654 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.594769 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.596481 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:55.596485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:55.596488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:55.596528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:55.598462 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.599030 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.599034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.599035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.599147 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:55.599535 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:55.599539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:55.599540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:55.599913 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:55.600114 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.600403 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.600560 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:09:55.604921 3201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:09:55.609016 3201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:09:55.630766 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.624913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:41643 (local address 127.0.0.1:40370)
0923 18:09:55.625320 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.625337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.625440 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.626250 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.626253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.626278 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:55.626399 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.626405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.627853 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.627856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.628832 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.628842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.630154 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.630158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.630265 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.630296 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.630399 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.630586 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260923 18:09:55.630770 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.623623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.18.44.193:36459)
0923 18:09:55.624380 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.624389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.624740 (+ 351us) server_negotiation.cc:407] Connection header received
0923 18:09:55.624842 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.624848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.625110 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.625279 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:55.625721 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.627035 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.628818 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.629580 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.629691 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.629843 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.630369 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.630548 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":150,"threads_started":1}
I20260923 18:09:55.631313 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.623555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:36459 (local address 127.0.0.1:59592)
0923 18:09:55.624522 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.624544 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.624645 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.625441 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.625445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.625459 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:55.625599 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.625607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.627212 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.627215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.628603 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.628618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.629780 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.629784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.629947 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.630098 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.630997 (+ 899us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.631127 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":76,"thread_start_us":180,"threads_started":1}
I20260923 18:09:55.631697 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.623333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40899 (local address 127.0.0.1:41650)
0923 18:09:55.624084 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.624162 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.624341 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.625225 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.625232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.625260 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:55.625465 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.625474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.626804 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.626808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.628603 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.628618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.631206 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.631210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.631366 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.631391 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.631466 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.631557 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":96,"thread_start_us":157,"threads_started":1}
I20260923 18:09:55.631799 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.624706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.18.44.194:41643)
0923 18:09:55.625633 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.625646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.625671 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:09:55.625750 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.625755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.625957 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.626269 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:55.626582 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.627695 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.629059 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.629962 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.630126 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.630302 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.631490 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.631644 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":169,"threads_started":1}
I20260923 18:09:55.632162 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.624070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.18.44.195:40899)
0923 18:09:55.624513 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.624527 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.624557 (+ 30us) server_negotiation.cc:407] Connection header received
0923 18:09:55.624668 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.624674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.625011 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.625212 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:55.625647 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.626642 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.629908 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.630778 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.630985 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.631134 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.631903 (+ 769us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.632026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":112,"threads_started":1}
I20260923 18:09:55.635146 3594 tablet_service.cc:1467] Processing CreateTablet for tablet d7dfec473e624b68958b40a96414ba1b (DEFAULT_TABLE table=test-table [id=82dd91ccde524eeaad28a6411a6dcd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:55.635536 3460 tablet_service.cc:1467] Processing CreateTablet for tablet d7dfec473e624b68958b40a96414ba1b (DEFAULT_TABLE table=test-table [id=82dd91ccde524eeaad28a6411a6dcd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:55.636492 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7dfec473e624b68958b40a96414ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:55.635536 3326 tablet_service.cc:1467] Processing CreateTablet for tablet d7dfec473e624b68958b40a96414ba1b (DEFAULT_TABLE table=test-table [id=82dd91ccde524eeaad28a6411a6dcd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:55.636912 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7dfec473e624b68958b40a96414ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:55.637122 3326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7dfec473e624b68958b40a96414ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:55.643810 3678 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Bootstrap starting.
I20260923 18:09:55.646651 3678 tablet_bootstrap.cc:654] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:55.647202 3679 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Bootstrap starting.
I20260923 18:09:55.649530 3678 log.cc:824] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:55.650153 3679 tablet_bootstrap.cc:654] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:55.650558 3680 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Bootstrap starting.
I20260923 18:09:55.651161 3679 log.cc:824] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:55.652649 3678 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: No bootstrap required, opened a new log
I20260923 18:09:55.652925 3678 ts_tablet_manager.cc:1397] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 18:09:55.653101 3680 tablet_bootstrap.cc:654] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:55.653945 3679 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: No bootstrap required, opened a new log
I20260923 18:09:55.654188 3679 ts_tablet_manager.cc:1397] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260923 18:09:55.654479 3680 log.cc:824] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:55.657027 3680 tablet_bootstrap.cc:492] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: No bootstrap required, opened a new log
I20260923 18:09:55.657277 3680 ts_tablet_manager.cc:1397] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:09:55.658452 3678 raft_consensus.cc:348] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.659053 3678 raft_consensus.cc:374] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:55.659135 3678 raft_consensus.cc:729] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e8facab36d4186aa9c51774163c1eb, State: Initialized, Role: FOLLOWER
I20260923 18:09:55.659087 3679 raft_consensus.cc:348] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.659498 3679 raft_consensus.cc:374] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:55.659566 3679 raft_consensus.cc:729] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc633c82c314b129bcff9ebf98b641e, State: Initialized, Role: FOLLOWER
I20260923 18:09:55.659956 3678 consensus_queue.cc:260] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.660127 3679 consensus_queue.cc:260] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.660981 3660 heartbeater.cc:503] Master 127.18.44.254:33297 was elected leader, sending a full tablet report...
I20260923 18:09:55.661386 3679 ts_tablet_manager.cc:1428] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260923 18:09:55.661362 3680 raft_consensus.cc:348] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.661670 3680 raft_consensus.cc:374] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:55.661731 3680 raft_consensus.cc:729] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027777d4c108455c8baf87a126b4e7b9, State: Initialized, Role: FOLLOWER
I20260923 18:09:55.661832 3392 heartbeater.cc:503] Master 127.18.44.254:33297 was elected leader, sending a full tablet report...
I20260923 18:09:55.662247 3680 consensus_queue.cc:260] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.663633 3678 ts_tablet_manager.cc:1428] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260923 18:09:55.663937 3526 heartbeater.cc:503] Master 127.18.44.254:33297 was elected leader, sending a full tablet report...
I20260923 18:09:55.664470 3680 ts_tablet_manager.cc:1428] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260923 18:09:55.704228 3393 tablet.cc:2369] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:09:55.761063 3527 tablet.cc:2369] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:55.797186 3685 raft_consensus.cc:482] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:55.797453 3685 raft_consensus.cc:504] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.798659 3685 leader_election.cc:288] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76e8facab36d4186aa9c51774163c1eb (127.18.44.195:40899), 027777d4c108455c8baf87a126b4e7b9 (127.18.44.194:41643)
I20260923 18:09:55.803303 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.799192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40899 (local address 127.18.44.193:34499)
0923 18:09:55.799610 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.799628 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.799783 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.800155 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.800160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.800204 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:55.800367 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.800377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.801134 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.801138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.802065 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.802076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.802781 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.802785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.802886 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.802909 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.802994 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.803089 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":191,"threads_started":1}
I20260923 18:09:55.803236 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.799342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:34499 (local address 127.18.44.195:40899)
0923 18:09:55.799466 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.799470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.799752 (+ 282us) server_negotiation.cc:407] Connection header received
0923 18:09:55.799924 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.799928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.799988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.800125 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:55.800499 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.801019 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.802214 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.802647 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.802764 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.802831 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.803009 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.803084 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 18:09:55.803807 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.799969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:38915 (local address 127.18.44.194:41643)
0923 18:09:55.800163 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.800169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.800313 (+ 144us) server_negotiation.cc:407] Connection header received
0923 18:09:55.800444 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.800448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.800560 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.800632 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:55.800936 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.801477 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.802578 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.803026 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.803150 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.803215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.803549 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.803651 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 18:09:55.804008 3614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7dfec473e624b68958b40a96414ba1b" candidate_uuid: "9dc633c82c314b129bcff9ebf98b641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8facab36d4186aa9c51774163c1eb" is_pre_election: true
I20260923 18:09:55.804265 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.799752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:41643 (local address 127.18.44.193:38915)
0923 18:09:55.800220 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.800234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:55.800338 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:55.800666 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:55.800669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:55.800685 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:55.800825 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.800830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.801593 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.801596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.802439 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:55.802449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.803155 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.803158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:55.803294 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.803319 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:09:55.804017 (+ 698us) client_negotiation.cc:240] Negotiation successful
0923 18:09:55.804101 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260923 18:09:55.804382 3614 raft_consensus.cc:2454] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc633c82c314b129bcff9ebf98b641e in term 0.
I20260923 18:09:55.804930 3281 leader_election.cc:302] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e8facab36d4186aa9c51774163c1eb, 9dc633c82c314b129bcff9ebf98b641e; no voters:
I20260923 18:09:55.805016 3480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7dfec473e624b68958b40a96414ba1b" candidate_uuid: "9dc633c82c314b129bcff9ebf98b641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027777d4c108455c8baf87a126b4e7b9" is_pre_election: true
I20260923 18:09:55.805239 3685 raft_consensus.cc:2781] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:09:55.805325 3685 raft_consensus.cc:482] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:09:55.805363 3685 raft_consensus.cc:3037] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:55.805413 3480 raft_consensus.cc:2454] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc633c82c314b129bcff9ebf98b641e in term 0.
I20260923 18:09:55.806638 3685 raft_consensus.cc:504] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.807008 3685 leader_election.cc:288] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [CANDIDATE]: Term 1 election: Requested vote from peers 76e8facab36d4186aa9c51774163c1eb (127.18.44.195:40899), 027777d4c108455c8baf87a126b4e7b9 (127.18.44.194:41643)
I20260923 18:09:55.807586 3614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7dfec473e624b68958b40a96414ba1b" candidate_uuid: "9dc633c82c314b129bcff9ebf98b641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8facab36d4186aa9c51774163c1eb"
I20260923 18:09:55.807586 3480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7dfec473e624b68958b40a96414ba1b" candidate_uuid: "9dc633c82c314b129bcff9ebf98b641e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027777d4c108455c8baf87a126b4e7b9"
I20260923 18:09:55.807796 3480 raft_consensus.cc:3037] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:55.807796 3614 raft_consensus.cc:3037] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:55.809450 3480 raft_consensus.cc:2454] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc633c82c314b129bcff9ebf98b641e in term 1.
I20260923 18:09:55.809450 3614 raft_consensus.cc:2454] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc633c82c314b129bcff9ebf98b641e in term 1.
W20260923 18:09:55.809957 3661 tablet.cc:2369] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:55.810032 3281 leader_election.cc:302] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e8facab36d4186aa9c51774163c1eb, 9dc633c82c314b129bcff9ebf98b641e; no voters:
I20260923 18:09:55.810328 3685 raft_consensus.cc:2781] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:55.810647 3685 raft_consensus.cc:686] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [term 1 LEADER]: Becoming Leader. State: Replica: 9dc633c82c314b129bcff9ebf98b641e, State: Running, Role: LEADER
I20260923 18:09:55.811043 3685 consensus_queue.cc:237] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } }
I20260923 18:09:55.815061 3201 catalog_manager.cc:5495] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dc633c82c314b129bcff9ebf98b641e (127.18.44.193). New cstate: current_term: 1 leader_uuid: "9dc633c82c314b129bcff9ebf98b641e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc633c82c314b129bcff9ebf98b641e" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 36459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 } health_report { overall_health: UNKNOWN } } }
I20260923 18:09:55.832290 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.827295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.18.44.193:36459)
0923 18:09:55.827513 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:55.827518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:55.827650 (+ 132us) server_negotiation.cc:407] Connection header received
0923 18:09:55.827823 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:55.827826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:55.827903 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:55.828026 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:09:55.828423 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.829085 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.830184 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:55.830440 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:55.830549 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:55.830961 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:09:55.831741 (+ 780us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:09:55.831873 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:55.831999 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 18:09:55.832084 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:09:55.840188 3480 raft_consensus.cc:1261] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 1 FOLLOWER]: Refusing update from remote peer 9dc633c82c314b129bcff9ebf98b641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:55.840188 3614 raft_consensus.cc:1261] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 1 FOLLOWER]: Refusing update from remote peer 9dc633c82c314b129bcff9ebf98b641e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:55.841143 3689 consensus_queue.cc:1035] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "027777d4c108455c8baf87a126b4e7b9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:55.841521 3685 consensus_queue.cc:1035] T d7dfec473e624b68958b40a96414ba1b P 9dc633c82c314b129bcff9ebf98b641e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e8facab36d4186aa9c51774163c1eb" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:55.852216 3694 mvcc.cc:205] Tried to move back new op lower bound from 7332605934951665664 to 7332605934844805120. Current Snapshot: MvccSnapshot[applied={T|T < 7332605934951665664}]
I20260923 18:09:55.868655 3201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:55.868922 3201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:55.872828 3201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878323d29b4a404c81ab698ae1c3d39d: Sending DeleteTablet for 3 replicas of tablet d7dfec473e624b68958b40a96414ba1b
I20260923 18:09:55.874182 3594 tablet_service.cc:1514] Processing DeleteTablet for tablet d7dfec473e624b68958b40a96414ba1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:41650
I20260923 18:09:55.874300 3326 tablet_service.cc:1514] Processing DeleteTablet for tablet d7dfec473e624b68958b40a96414ba1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:59592
I20260923 18:09:55.874300 3460 tablet_service.cc:1514] Processing DeleteTablet for tablet d7dfec473e624b68958b40a96414ba1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:55 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:40370
I20260923 18:09:55.875227 3701 tablet_replica.cc:330] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: stopping tablet replica
I20260923 18:09:55.875339 3702 tablet_replica.cc:330] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: stopping tablet replica
I20260923 18:09:55.875792 3701 raft_consensus.cc:2229] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:55.875952 3702 raft_consensus.cc:2229] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:55.876312 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3263
I20260923 18:09:55.876472 3702 raft_consensus.cc:2258] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:55.877487 3701 raft_consensus.cc:2258] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:55.878502 3702 ts_tablet_manager.cc:1905] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:55.879549 3701 ts_tablet_manager.cc:1905] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:55.881660 3702 ts_tablet_manager.cc:1918] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:55.881783 3702 log.cc:1196] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/d7dfec473e624b68958b40a96414ba1b
I20260923 18:09:55.882071 3702 ts_tablet_manager.cc:1939] T d7dfec473e624b68958b40a96414ba1b P 027777d4c108455c8baf87a126b4e7b9: Deleting consensus metadata
I20260923 18:09:55.883183 3701 ts_tablet_manager.cc:1918] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:55.883383 3701 log.cc:1196] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/d7dfec473e624b68958b40a96414ba1b
I20260923 18:09:55.883643 3187 catalog_manager.cc:4841] TS 027777d4c108455c8baf87a126b4e7b9 (127.18.44.194:41643): tablet d7dfec473e624b68958b40a96414ba1b (table test-table [id=82dd91ccde524eeaad28a6411a6dcd95]) successfully deleted
I20260923 18:09:55.883741 3701 ts_tablet_manager.cc:1939] T d7dfec473e624b68958b40a96414ba1b P 76e8facab36d4186aa9c51774163c1eb: Deleting consensus metadata
I20260923 18:09:55.885725 3187 catalog_manager.cc:4841] TS 76e8facab36d4186aa9c51774163c1eb (127.18.44.195:40899): tablet d7dfec473e624b68958b40a96414ba1b (table test-table [id=82dd91ccde524eeaad28a6411a6dcd95]) successfully deleted
W20260923 18:09:55.887049 3185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:36459 (error 108)
I20260923 18:09:55.887146 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3397
I20260923 18:09:55.889753 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:55.889338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:36459 (local address 127.0.0.1:59606)
0923 18:09:55.889541 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:55.889638 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:36459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 18:09:55.889957 3185 catalog_manager.cc:4584] TS 9dc633c82c314b129bcff9ebf98b641e (127.18.44.193:36459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7dfec473e624b68958b40a96414ba1b: Network error: Client connection negotiation failed: client connection to 127.18.44.193:36459: connect: Connection refused (error 111)
I20260923 18:09:55.893532 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3531
I20260923 18:09:55.899694 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3169
2026-09-23T18:09:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2734 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T18:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:09:55Z Disabled control of system clock
I20260923 18:09:55.937398 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:42825
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:09:56.087455 3708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:56.087906 3708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:56.087953 3708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:56.092844 3708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:56.092958 3708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:56.092972 3708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:56.092991 3708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:56.093007 3708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:56.100242 3708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:42825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:56.102042 3708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:56.104063 3708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:56.115082 3713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:56.115082 3716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.115336 3708 server_base.cc:1034] running on GCE node
W20260923 18:09:56.115109 3714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.116195 3708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:56.117470 3708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:56.120888 3708 hybrid_clock.cc:648] HybridClock initialized: now 1790186996120725 us; error 167 us; skew 500 ppm
I20260923 18:09:56.123802 3708 webserver.cc:460] Webserver started at http://127.18.44.254:36267/ using document root <none> and password file <none>
I20260923 18:09:56.124694 3708 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:56.124778 3708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:56.125008 3708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:56.127470 3708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "4663db8dc9f04bd4b2a8f79baf301a6e"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d291a23fbeaef994f0ee32e300d737df"
server_key_iv: "db9666953a95f1d3e967d2b19d98dce0"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.128099 3708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "4663db8dc9f04bd4b2a8f79baf301a6e"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d291a23fbeaef994f0ee32e300d737df"
server_key_iv: "db9666953a95f1d3e967d2b19d98dce0"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.132890 3708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:09:56.136264 3722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.138207 3708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:56.138384 3708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "4663db8dc9f04bd4b2a8f79baf301a6e"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d291a23fbeaef994f0ee32e300d737df"
server_key_iv: "db9666953a95f1d3e967d2b19d98dce0"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.138525 3708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:56.150722 3708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:56.151589 3708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:56.151782 3708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:56.159575 3708 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:42825
I20260923 18:09:56.159590 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:42825 every 8 connection(s)
I20260923 18:09:56.160686 3708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:09:56.165570 3775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:56.168026 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3708
I20260923 18:09:56.168208 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:09:56.168561 18611 external_mini_cluster.cc:1428] Setting key f8bb88159484d3bedac418c92afd1df5
I20260923 18:09:56.174487 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: Bootstrap starting.
I20260923 18:09:56.177935 3775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:56.179136 3775 log.cc:824] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:56.181844 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.170635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.18.44.254:42825)
0923 18:09:56.171801 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.171824 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.171878 (+ 54us) server_negotiation.cc:407] Connection header received
0923 18:09:56.173151 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.173184 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.173806 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.174171 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:56.174747 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.175952 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.177382 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.177986 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.178205 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.179499 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:56.179505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:56.179528 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:09:56.180147 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:56.180249 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:09:56.180292 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:56.180407 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.180647 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.180975 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":209,"threads_started":1}
I20260923 18:09:56.182292 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: No bootstrap required, opened a new log
I20260923 18:09:56.187565 3775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } }
I20260923 18:09:56.187988 3775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:56.188045 3775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4663db8dc9f04bd4b2a8f79baf301a6e, State: Initialized, Role: FOLLOWER
I20260923 18:09:56.188575 3775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } }
I20260923 18:09:56.188736 3775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:09:56.188810 3775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:09:56.188925 3775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:56.190219 3775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } }
I20260923 18:09:56.190681 3775 leader_election.cc:302] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4663db8dc9f04bd4b2a8f79baf301a6e; no voters:
I20260923 18:09:56.191113 3775 leader_election.cc:288] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:09:56.191375 3780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:56.191794 3780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [term 1 LEADER]: Becoming Leader. State: Replica: 4663db8dc9f04bd4b2a8f79baf301a6e, State: Running, Role: LEADER
I20260923 18:09:56.192297 3780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } }
I20260923 18:09:56.192929 3775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:09:56.194475 3781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } } }
I20260923 18:09:56.194520 3782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4663db8dc9f04bd4b2a8f79baf301a6e. Latest consensus state: current_term: 1 leader_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4663db8dc9f04bd4b2a8f79baf301a6e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 42825 } } }
I20260923 18:09:56.194707 3782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:56.194706 3781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:56.195688 3790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:09:56.200014 3790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:09:56.208369 3790 catalog_manager.cc:1347] Generated new cluster ID: 7d1a4c1fb45f42bdb078d5f349f0e4b2
I20260923 18:09:56.208508 3790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:09:56.224797 3790 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:09:56.226347 3790 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:09:56.233379 3790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: Generated new TSK 0
I20260923 18:09:56.234496 3790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:09:56.238909 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--builtin_ntp_servers=127.18.44.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:09:56.382283 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:56.382722 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:56.382792 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:56.388684 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:56.388796 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:56.388893 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:09:56.396651 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:56.398212 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:56.400940 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:56.414588 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:56.414588 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:56.414597 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.415021 3799 server_base.cc:1034] running on GCE node
I20260923 18:09:56.415812 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:56.416829 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:56.418061 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790186996418037 us; error 48 us; skew 500 ppm
I20260923 18:09:56.420993 3799 webserver.cc:460] Webserver started at http://127.18.44.193:35657/ using document root <none> and password file <none>
I20260923 18:09:56.422056 3799 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:56.422226 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:56.422485 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:56.426146 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "3481242a3bc34546b4988e841d238461"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "106577b71cb44fe2b3a23f7c15bb7c3c"
server_key_iv: "cd8fe6ecc554c8795d974b4d9b599a2d"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.427058 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "3481242a3bc34546b4988e841d238461"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "106577b71cb44fe2b3a23f7c15bb7c3c"
server_key_iv: "cd8fe6ecc554c8795d974b4d9b599a2d"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.434176 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260923 18:09:56.438661 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.441145 3799 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260923 18:09:56.441434 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "3481242a3bc34546b4988e841d238461"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "106577b71cb44fe2b3a23f7c15bb7c3c"
server_key_iv: "cd8fe6ecc554c8795d974b4d9b599a2d"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.441632 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:56.466315 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:56.467381 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:56.467607 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:56.468413 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:56.469722 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:56.469776 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.469842 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:56.469872 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.481688 3799 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:45635
I20260923 18:09:56.481761 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:45635 every 8 connection(s)
I20260923 18:09:56.483180 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:09:56.488880 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3799
I20260923 18:09:56.489028 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:09:56.489368 18611 external_mini_cluster.cc:1428] Setting key 3a4f5d9d369e65c8998815563f915616
I20260923 18:09:56.491503 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--builtin_ntp_servers=127.18.44.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:09:56.493948 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.485446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:35575 (local address 127.18.44.254:42825)
0923 18:09:56.485605 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.485610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.486960 (+ 1350us) server_negotiation.cc:407] Connection header received
0923 18:09:56.488548 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.488552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.488615 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.488770 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:56.489911 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.490596 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.491932 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.492168 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.492260 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.493453 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:56.493457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:56.493460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:09:56.493473 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:56.493494 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:09:56.493502 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:56.493560 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.493708 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.493800 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:09:56.494961 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.485888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42825 (local address 127.18.44.193:35575)
0923 18:09:56.486779 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.486827 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.488271 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.488934 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.488942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.489055 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:56.489750 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.489766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.490728 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.490732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.491807 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.491820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.492335 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.492339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.492505 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.493166 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:09:56.493168 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:56.493212 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:09:56.493295 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:56.493572 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:56.493581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:56.493587 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.493702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.494073 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":152,"threads_started":1}
I20260923 18:09:56.496315 3927 heartbeater.cc:347] Connected to a master server at 127.18.44.254:42825
I20260923 18:09:56.496634 3927 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:56.497452 3927 heartbeater.cc:511] Master 127.18.44.254:42825 requested a full tablet report, sending...
I20260923 18:09:56.499725 3735 ts_manager.cc:194] Registered new tserver with Master: 3481242a3bc34546b4988e841d238461 (127.18.44.193:45635)
I20260923 18:09:56.501055 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.193:35575
W20260923 18:09:56.615558 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:56.615869 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:56.615922 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:56.620375 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:56.620465 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:56.620550 3930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:09:56.626916 3930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:56.628543 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:56.630174 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:56.641681 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:56.641696 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.641810 3930 server_base.cc:1034] running on GCE node
W20260923 18:09:56.641708 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.642549 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:56.643437 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:56.644656 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790186996644635 us; error 40 us; skew 500 ppm
I20260923 18:09:56.647141 3930 webserver.cc:460] Webserver started at http://127.18.44.194:36479/ using document root <none> and password file <none>
I20260923 18:09:56.647981 3930 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:56.648072 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:56.648341 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:56.651371 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "286cabbc2a014f37a811f057f10954bc"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "5cfcd1188847725c90cd8d77c204b0b9"
server_key_iv: "bd09c9de7ce42a1769ee81e88765bdfd"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.652091 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "286cabbc2a014f37a811f057f10954bc"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "5cfcd1188847725c90cd8d77c204b0b9"
server_key_iv: "bd09c9de7ce42a1769ee81e88765bdfd"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.657058 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 18:09:56.660605 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.662514 3930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:09:56.662688 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "286cabbc2a014f37a811f057f10954bc"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "5cfcd1188847725c90cd8d77c204b0b9"
server_key_iv: "bd09c9de7ce42a1769ee81e88765bdfd"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.662830 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:56.692021 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:56.692879 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:56.693074 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:56.693784 3930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:56.695542 3930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:56.695617 3930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.695698 3930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:56.695721 3930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.706373 3930 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:44063
I20260923 18:09:56.706526 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:44063 every 8 connection(s)
I20260923 18:09:56.707423 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:09:56.709368 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 3930
I20260923 18:09:56.709487 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:09:56.709803 18611 external_mini_cluster.cc:1428] Setting key 76d6fb32a26d5876bae7a75de82e9a93
I20260923 18:09:56.711648 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--builtin_ntp_servers=127.18.44.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:09:56.718319 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.709669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:52287 (local address 127.18.44.254:42825)
0923 18:09:56.709835 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.709841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.711026 (+ 1185us) server_negotiation.cc:407] Connection header received
0923 18:09:56.712652 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.712655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.712714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.712786 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:56.713719 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.714267 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.715927 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.716213 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.716400 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.717792 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:56.717795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:56.717797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:09:56.717813 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:56.717836 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:09:56.717844 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:56.717926 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.718114 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.718190 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:09:56.719995 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42825 (local address 127.18.44.194:52287)
0923 18:09:56.710872 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.710929 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.712427 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.712973 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.712980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.713069 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:56.713582 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.713595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.714402 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.714406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.715764 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.715779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.716410 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.716415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.716646 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.717405 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:09:56.717407 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:56.717475 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:09:56.717583 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:56.717941 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:56.717951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:56.717957 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.718110 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.718501 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":127,"threads_started":1}
I20260923 18:09:56.721668 4058 heartbeater.cc:347] Connected to a master server at 127.18.44.254:42825
I20260923 18:09:56.721943 4058 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:56.722543 4058 heartbeater.cc:511] Master 127.18.44.254:42825 requested a full tablet report, sending...
I20260923 18:09:56.723843 3735 ts_manager.cc:194] Registered new tserver with Master: 286cabbc2a014f37a811f057f10954bc (127.18.44.194:44063)
I20260923 18:09:56.724557 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.194:52287
W20260923 18:09:56.836215 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:56.836721 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:56.836782 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:56.841194 4061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:56.841284 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:56.841346 4061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:09:56.846479 4061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:56.847671 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:56.849164 4061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:56.859124 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:56.859128 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.859170 4061 server_base.cc:1034] running on GCE node
W20260923 18:09:56.859128 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:56.859848 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:56.860493 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:56.861678 4061 hybrid_clock.cc:648] HybridClock initialized: now 1790186996861632 us; error 61 us; skew 500 ppm
I20260923 18:09:56.863879 4061 webserver.cc:460] Webserver started at http://127.18.44.195:43405/ using document root <none> and password file <none>
I20260923 18:09:56.864470 4061 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:56.864543 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:56.864735 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:56.866808 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "37718b474da34534a2df4ecffd1c8194"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d8f11d8a136135509c1569b0f4320879"
server_key_iv: "481b4921c2d0ffcecb492a53c3b915cb"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.867442 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "37718b474da34534a2df4ecffd1c8194"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d8f11d8a136135509c1569b0f4320879"
server_key_iv: "481b4921c2d0ffcecb492a53c3b915cb"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.871126 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:09:56.873732 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.875211 4061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:09:56.875456 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "37718b474da34534a2df4ecffd1c8194"
format_stamp: "Formatted at 2026-09-23 18:09:56 on dist-test-slave-rvjj"
server_key: "d8f11d8a136135509c1569b0f4320879"
server_key_iv: "481b4921c2d0ffcecb492a53c3b915cb"
server_key_version: "encryptionkey@0"
I20260923 18:09:56.875591 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:56.892621 4061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:56.893404 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:56.893585 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:56.894256 4061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:56.895308 4061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:56.895367 4061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.895414 4061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:56.895443 4061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:56.904232 4061 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40215
I20260923 18:09:56.904264 4189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40215 every 8 connection(s)
I20260923 18:09:56.905236 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:09:56.909905 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4061
I20260923 18:09:56.910002 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:09:56.910243 18611 external_mini_cluster.cc:1428] Setting key f2db37a0394b1f7ab63f439ade182253
I20260923 18:09:56.915925 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.909285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:39121 (local address 127.18.44.254:42825)
0923 18:09:56.909423 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.909427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.909441 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:09:56.910717 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.910721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.910775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.910849 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:56.912001 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.912614 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.913905 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.914138 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.914231 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.915493 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:56.915497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:56.915499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:09:56.915514 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:56.915536 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:09:56.915544 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:56.915612 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.915733 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.915795 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:09:56.916885 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.907914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:42825 (local address 127.18.44.195:39121)
0923 18:09:56.908969 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.909030 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.910495 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.911037 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.911047 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.911163 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:56.911842 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.911864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.912784 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.912787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.913759 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.913770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.914306 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.914310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.914504 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.915161 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:09:56.915163 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:56.915208 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:09:56.915324 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:56.915619 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:56.915624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:56.915628 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.915718 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.916017 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":151,"threads_started":1}
I20260923 18:09:56.918116 4190 heartbeater.cc:347] Connected to a master server at 127.18.44.254:42825
I20260923 18:09:56.918359 4190 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:56.918908 4190 heartbeater.cc:511] Master 127.18.44.254:42825 requested a full tablet report, sending...
I20260923 18:09:56.919979 3735 ts_manager.cc:194] Registered new tserver with Master: 37718b474da34534a2df4ecffd1c8194 (127.18.44.195:40215)
I20260923 18:09:56.920564 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.195:39121
I20260923 18:09:56.924425 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:09:56.930555 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.926674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.18.44.254:42825)
0923 18:09:56.926852 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.926856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.926989 (+ 133us) server_negotiation.cc:407] Connection header received
0923 18:09:56.927037 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.927039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.927083 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.927157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:56.927528 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.928054 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.929143 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.929350 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.929469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.930164 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:56.930167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:56.930169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:09:56.930182 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:56.930199 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:09:56.930206 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:56.930262 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.930368 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.930443 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 18:09:56.933595 3735 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:09:56.936858 3735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:09:56.952862 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.946107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:44063 (local address 127.0.0.1:45438)
0923 18:09:56.946513 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.946583 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.946762 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.948094 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.948100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.948141 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:56.948341 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.948349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949840 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.949844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.951142 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.951149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.952262 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.952499 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952522 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.952617 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.952682 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":224,"spinlock_wait_cycles":5248,"thread_start_us":110,"threads_started":1}
I20260923 18:09:56.952893 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.946305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.18.44.194:44063)
0923 18:09:56.947528 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.947542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.947570 (+ 28us) server_negotiation.cc:407] Connection header received
0923 18:09:56.947652 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.947658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.947876 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.948090 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:56.948474 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949721 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.951285 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.952119 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952313 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952503 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.952649 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.952743 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":210,"threads_started":1}
I20260923 18:09:56.952741 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.946495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45635 (local address 127.0.0.1:52258)
0923 18:09:56.947088 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.947102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.947200 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.948095 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.948101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.948139 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:56.948600 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.948607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949581 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.949585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.951119 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.951132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.952078 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.952206 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952248 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.952395 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.952540 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":330,"spinlock_wait_cycles":12928,"thread_start_us":126,"threads_started":1}
I20260923 18:09:56.953267 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.947342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.18.44.195:40215)
0923 18:09:56.947891 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.947903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.947926 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:09:56.948001 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.948005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.948194 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.948346 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:56.948598 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949721 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.951314 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.952118 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952444 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952602 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.952946 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.953121 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":139,"threads_started":1}
I20260923 18:09:56.953642 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.946904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40215 (local address 127.0.0.1:58540)
0923 18:09:56.947102 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:56.947114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:56.947200 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:56.948338 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:56.948341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:56.948357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:56.948457 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.948465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949875 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.949878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.951152 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:56.951161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.952262 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:56.952785 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952810 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:09:56.952888 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 18:09:56.953542 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":396,"spinlock_wait_cycles":5632,"thread_start_us":93,"threads_started":1}
I20260923 18:09:56.956037 4123 tablet_service.cc:1467] Processing CreateTablet for tablet f02eb06bcd714975affb49773aacdc89 (DEFAULT_TABLE table=test-table [id=dee7b858ed5449b696e22d45c0e5675a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:56.955998 3992 tablet_service.cc:1467] Processing CreateTablet for tablet f02eb06bcd714975affb49773aacdc89 (DEFAULT_TABLE table=test-table [id=dee7b858ed5449b696e22d45c0e5675a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:56.956959 4123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02eb06bcd714975affb49773aacdc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:56.956959 3992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02eb06bcd714975affb49773aacdc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:56.963177 4207 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Bootstrap starting.
I20260923 18:09:56.963608 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:56.947198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.18.44.193:45635)
0923 18:09:56.947545 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:56.947551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:56.947572 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:09:56.947652 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:56.947658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:56.947876 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:56.948324 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:56.948693 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.949458 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.951331 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:56.951941 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:56.952062 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:56.952310 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:56.953808 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0923 18:09:56.963413 (+ 9605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":131,"threads_started":1}
I20260923 18:09:56.964613 4208 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Bootstrap starting.
I20260923 18:09:56.965710 4207 tablet_bootstrap.cc:654] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:56.966416 4207 log.cc:824] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:56.966741 4208 tablet_bootstrap.cc:654] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:56.966706 3861 tablet_service.cc:1467] Processing CreateTablet for tablet f02eb06bcd714975affb49773aacdc89 (DEFAULT_TABLE table=test-table [id=dee7b858ed5449b696e22d45c0e5675a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:56.967590 4208 log.cc:824] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:56.967912 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02eb06bcd714975affb49773aacdc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:56.968554 4207 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: No bootstrap required, opened a new log
I20260923 18:09:56.968742 4207 ts_tablet_manager.cc:1397] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:09:56.969857 4208 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: No bootstrap required, opened a new log
I20260923 18:09:56.970054 4208 ts_tablet_manager.cc:1397] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:09:56.972330 4207 raft_consensus.cc:348] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.972606 4207 raft_consensus.cc:374] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:56.972662 4207 raft_consensus.cc:729] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286cabbc2a014f37a811f057f10954bc, State: Initialized, Role: FOLLOWER
I20260923 18:09:56.973119 4207 consensus_queue.cc:260] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.973809 4058 heartbeater.cc:503] Master 127.18.44.254:42825 was elected leader, sending a full tablet report...
I20260923 18:09:56.974161 4207 ts_tablet_manager.cc:1428] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:09:56.974483 4208 raft_consensus.cc:348] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.974869 4208 raft_consensus.cc:374] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:56.974937 4208 raft_consensus.cc:729] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37718b474da34534a2df4ecffd1c8194, State: Initialized, Role: FOLLOWER
I20260923 18:09:56.975414 4208 consensus_queue.cc:260] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.976097 4212 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Bootstrap starting.
I20260923 18:09:56.976449 4208 ts_tablet_manager.cc:1428] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:09:56.976804 4190 heartbeater.cc:503] Master 127.18.44.254:42825 was elected leader, sending a full tablet report...
I20260923 18:09:56.979665 4212 tablet_bootstrap.cc:654] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:56.980485 4212 log.cc:824] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:56.982627 4212 tablet_bootstrap.cc:492] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: No bootstrap required, opened a new log
I20260923 18:09:56.982846 4212 ts_tablet_manager.cc:1397] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 18:09:56.986173 4212 raft_consensus.cc:348] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.986436 4212 raft_consensus.cc:374] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:56.986495 4212 raft_consensus.cc:729] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3481242a3bc34546b4988e841d238461, State: Initialized, Role: FOLLOWER
I20260923 18:09:56.986904 4212 consensus_queue.cc:260] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:56.987687 3927 heartbeater.cc:503] Master 127.18.44.254:42825 was elected leader, sending a full tablet report...
I20260923 18:09:56.987998 4212 ts_tablet_manager.cc:1428] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 18:09:57.156316 4191 tablet.cc:2369] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:57.167308 4213 raft_consensus.cc:482] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:57.167553 4213 raft_consensus.cc:504] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:57.168758 4213 leader_election.cc:288] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 286cabbc2a014f37a811f057f10954bc (127.18.44.194:44063), 3481242a3bc34546b4988e841d238461 (127.18.44.193:45635)
I20260923 18:09:57.172760 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.169009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:44063 (local address 127.18.44.195:50607)
0923 18:09:57.169206 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:57.169220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:57.169318 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:57.169587 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:57.169590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:57.169607 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:57.169727 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:57.169733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.170619 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.170622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:57.171494 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:57.171503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.172353 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.172357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:57.172459 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:57.172484 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:09:57.172552 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:09:57.172622 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 18:09:57.172780 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.169180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50607 (local address 127.18.44.194:44063)
0923 18:09:57.169309 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:57.169313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:57.169329 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:09:57.169423 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:57.169426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:57.169478 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:57.169570 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:57.169837 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.170521 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.171625 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.172238 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.172349 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:57.172400 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:57.172574 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 18:09:57.172650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:09:57.173408 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.169138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:35647 (local address 127.18.44.193:45635)
0923 18:09:57.169262 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:57.169267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:57.169829 (+ 562us) server_negotiation.cc:407] Connection header received
0923 18:09:57.170024 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:57.170027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:57.170076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:57.170140 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:57.170408 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.171158 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.172248 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.172809 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.172922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:57.172966 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:57.173233 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 18:09:57.173317 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:09:57.173416 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.169008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45635 (local address 127.18.44.195:35647)
0923 18:09:57.169741 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:57.169752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:57.169910 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:57.170171 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:57.170174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:57.170190 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:57.170314 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:57.170320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.171298 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.171302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:57.172123 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:57.172132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.173031 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.173035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:57.173126 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:57.173148 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:09:57.173206 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 18:09:57.173268 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":282,"thread_start_us":142,"threads_started":1}
I20260923 18:09:57.173557 4012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02eb06bcd714975affb49773aacdc89" candidate_uuid: "37718b474da34534a2df4ecffd1c8194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286cabbc2a014f37a811f057f10954bc" is_pre_election: true
I20260923 18:09:57.173935 4012 raft_consensus.cc:2454] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37718b474da34534a2df4ecffd1c8194 in term 0.
I20260923 18:09:57.173935 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02eb06bcd714975affb49773aacdc89" candidate_uuid: "37718b474da34534a2df4ecffd1c8194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3481242a3bc34546b4988e841d238461" is_pre_election: true
I20260923 18:09:57.174173 3881 raft_consensus.cc:2454] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37718b474da34534a2df4ecffd1c8194 in term 0.
I20260923 18:09:57.174530 4079 leader_election.cc:302] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286cabbc2a014f37a811f057f10954bc, 37718b474da34534a2df4ecffd1c8194; no voters:
I20260923 18:09:57.174839 4213 raft_consensus.cc:2781] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:09:57.174935 4213 raft_consensus.cc:482] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:09:57.174981 4213 raft_consensus.cc:3037] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:57.176132 4213 raft_consensus.cc:504] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:57.176499 4213 leader_election.cc:288] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [CANDIDATE]: Term 1 election: Requested vote from peers 286cabbc2a014f37a811f057f10954bc (127.18.44.194:44063), 3481242a3bc34546b4988e841d238461 (127.18.44.193:45635)
I20260923 18:09:57.176859 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02eb06bcd714975affb49773aacdc89" candidate_uuid: "37718b474da34534a2df4ecffd1c8194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3481242a3bc34546b4988e841d238461"
I20260923 18:09:57.176867 4012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02eb06bcd714975affb49773aacdc89" candidate_uuid: "37718b474da34534a2df4ecffd1c8194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286cabbc2a014f37a811f057f10954bc"
I20260923 18:09:57.176980 3881 raft_consensus.cc:3037] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:57.177093 4012 raft_consensus.cc:3037] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:57.178027 3881 raft_consensus.cc:2454] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37718b474da34534a2df4ecffd1c8194 in term 1.
I20260923 18:09:57.178198 4012 raft_consensus.cc:2454] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37718b474da34534a2df4ecffd1c8194 in term 1.
I20260923 18:09:57.178354 4077 leader_election.cc:302] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3481242a3bc34546b4988e841d238461, 37718b474da34534a2df4ecffd1c8194; no voters:
I20260923 18:09:57.178534 4213 raft_consensus.cc:2781] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:57.178754 4213 raft_consensus.cc:686] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 1 LEADER]: Becoming Leader. State: Replica: 37718b474da34534a2df4ecffd1c8194, State: Running, Role: LEADER
I20260923 18:09:57.179123 4213 consensus_queue.cc:237] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } }
I20260923 18:09:57.182418 3734 catalog_manager.cc:5495] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37718b474da34534a2df4ecffd1c8194 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "37718b474da34534a2df4ecffd1c8194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37718b474da34534a2df4ecffd1c8194" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 } health_report { overall_health: UNKNOWN } } }
W20260923 18:09:57.208719 4059 tablet.cc:2369] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:09:57.234827 3928 tablet.cc:2369] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:57.248510 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.244741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.18.44.195:40215)
0923 18:09:57.244891 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:57.244895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:57.244911 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:09:57.245008 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:57.245010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:57.245068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:57.245160 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:09:57.245404 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.245929 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.246845 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:57.247072 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:57.247163 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:57.247498 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:09:57.247944 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:09:57.248052 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:57.248189 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 18:09:57.248287 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:09:57.255055 4012 raft_consensus.cc:1261] T f02eb06bcd714975affb49773aacdc89 P 286cabbc2a014f37a811f057f10954bc [term 1 FOLLOWER]: Refusing update from remote peer 37718b474da34534a2df4ecffd1c8194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:57.255055 3881 raft_consensus.cc:1261] T f02eb06bcd714975affb49773aacdc89 P 3481242a3bc34546b4988e841d238461 [term 1 FOLLOWER]: Refusing update from remote peer 37718b474da34534a2df4ecffd1c8194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:57.255795 4217 consensus_queue.cc:1035] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3481242a3bc34546b4988e841d238461" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:57.255992 4213 consensus_queue.cc:1035] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286cabbc2a014f37a811f057f10954bc" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 44063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:57.264860 4221 mvcc.cc:205] Tried to move back new op lower bound from 7332605940748398592 to 7332605940448329728. Current Snapshot: MvccSnapshot[applied={T|T < 7332605940748398592}]
I20260923 18:09:57.273700 3734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:57.273934 3734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:57.276726 3734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4663db8dc9f04bd4b2a8f79baf301a6e: Sending DeleteTablet for 3 replicas of tablet f02eb06bcd714975affb49773aacdc89
I20260923 18:09:57.277714 3992 tablet_service.cc:1514] Processing DeleteTablet for tablet f02eb06bcd714975affb49773aacdc89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:57 UTC) from {username='slave'} at 127.0.0.1:45438
I20260923 18:09:57.277760 4123 tablet_service.cc:1514] Processing DeleteTablet for tablet f02eb06bcd714975affb49773aacdc89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:57 UTC) from {username='slave'} at 127.0.0.1:58540
I20260923 18:09:57.278319 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet f02eb06bcd714975affb49773aacdc89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:57 UTC) from {username='slave'} at 127.0.0.1:52258
I20260923 18:09:57.278442 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3799
I20260923 18:09:57.283613 4231 tablet_replica.cc:330] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: stopping tablet replica
I20260923 18:09:57.284041 4231 raft_consensus.cc:2229] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:09:57.284561 4231 raft_consensus.cc:2258] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:09:57.285038 3724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:45635 (error 108)
I20260923 18:09:57.285147 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3930
I20260923 18:09:57.286825 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.286511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:45635 (local address 127.0.0.1:52270)
0923 18:09:57.286648 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:57.286722 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:45635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 18:09:57.287393 3724 catalog_manager.cc:4584] TS 3481242a3bc34546b4988e841d238461 (127.18.44.193:45635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f02eb06bcd714975affb49773aacdc89: Network error: Client connection negotiation failed: client connection to 127.18.44.193:45635: connect: Connection refused (error 111)
I20260923 18:09:57.288791 4231 ts_tablet_manager.cc:1905] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:57.291501 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:57.291183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:44063 (local address 127.0.0.1:45442)
0923 18:09:57.291358 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:57.291411 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:44063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 18:09:57.291611 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4061
I20260923 18:09:57.291774 4231 ts_tablet_manager.cc:1918] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:57.291885 4231 log.cc:1196] T f02eb06bcd714975affb49773aacdc89 P 37718b474da34534a2df4ecffd1c8194: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/f02eb06bcd714975affb49773aacdc89
I20260923 18:09:57.297118 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 3708
2026-09-23T18:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1392 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:09:57 dist-test-slave-rvjj krb5kdc[4235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:09:57 dist-test-slave-rvjj krb5kdc[4235](info): set up 2 sockets
Sep 23 18:09:57 dist-test-slave-rvjj krb5kdc[4235](info): commencing operation
krb5kdc: starting...
W20260923 18:09:58.272382 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:09:58 dist-test-slave-rvjj krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:09:58.426884 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41877
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:58.554636 4248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:58.554965 4248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:58.555009 4248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:58.559412 4248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:58.559502 4248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:58.559521 4248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:58.559538 4248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:58.559552 4248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:58.565516 4248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:58.566676 4248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:58.567801 4248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:58.578943 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:58.579051 4248 server_base.cc:1034] running on GCE node
W20260923 18:09:58.578871 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:58.579216 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:58.579968 4248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:58.581205 4248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:58.582417 4248 hybrid_clock.cc:648] HybridClock initialized: now 1790186998582394 us; error 40 us; skew 500 ppm
Sep 23 18:09:58 dist-test-slave-rvjj krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:58.585577 4248 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:09:58.586678 4248 webserver.cc:460] Webserver started at http://127.18.44.254:46365/ using document root <none> and password file <none>
I20260923 18:09:58.587296 4248 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:58.587374 4248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:58.587581 4248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:58.589761 4248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "54015cbd84da4e0aa48d8d447ed1c082"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "5f22942e52b780f15641742503fba42e"
server_key_iv: "a4779d7ee13366d28b9c54b442b8e26f"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.590353 4248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "54015cbd84da4e0aa48d8d447ed1c082"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "5f22942e52b780f15641742503fba42e"
server_key_iv: "a4779d7ee13366d28b9c54b442b8e26f"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.594858 4248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:09:58.597916 4263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:58.599526 4248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:09:58.599679 4248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "54015cbd84da4e0aa48d8d447ed1c082"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "5f22942e52b780f15641742503fba42e"
server_key_iv: "a4779d7ee13366d28b9c54b442b8e26f"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.599817 4248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:58.613407 4248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:58.617183 4248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:58.617489 4248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:58.625881 4248 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:41877
I20260923 18:09:58.625914 4315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:41877 every 8 connection(s)
I20260923 18:09:58.626940 4248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:09:58.630373 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:58.634907 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4248
I20260923 18:09:58.635066 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:09:58.635416 18611 external_mini_cluster.cc:1428] Setting key 7508be04789daadb7c6b5e0f29d18e04
I20260923 18:09:58.637102 4316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: Bootstrap starting.
I20260923 18:09:58.640215 4316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:58.641057 4316 log.cc:824] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:58.643208 4316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: No bootstrap required, opened a new log
Sep 23 18:09:58 dist-test-slave-rvjj krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:58.648109 4316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } }
I20260923 18:09:58.648480 4316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:58.648533 4316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54015cbd84da4e0aa48d8d447ed1c082, State: Initialized, Role: FOLLOWER
I20260923 18:09:58.649209 4316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } }
I20260923 18:09:58.649420 4316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:09:58.649511 4316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:09:58.649618 4316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:58.651219 4316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } }
I20260923 18:09:58.651724 4316 leader_election.cc:302] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54015cbd84da4e0aa48d8d447ed1c082; no voters:
I20260923 18:09:58.652113 4316 leader_election.cc:288] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:09:58.652303 4321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:58.652510 4321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [term 1 LEADER]: Becoming Leader. State: Replica: 54015cbd84da4e0aa48d8d447ed1c082, State: Running, Role: LEADER
I20260923 18:09:58.652792 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:58.637089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.18.44.254:41877)
0923 18:09:58.637595 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:58.637615 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:58.637661 (+ 46us) server_negotiation.cc:407] Connection header received
0923 18:09:58.638689 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:58.638715 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:58.639191 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:58.639646 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:58.640447 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.641543 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.642439 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.642968 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.643128 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:58.646329 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:58.646341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:58.646366 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:58.646417 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:58.648664 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:58.649402 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:58.649410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:58.649417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:58.649586 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:58.650020 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:58.650024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:58.650026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:58.650682 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:58.650915 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:58.651224 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 18:09:58.651658 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":168,"threads_started":1}
I20260923 18:09:58.652884 4321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } }
I20260923 18:09:58.653535 4316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:09:58.654249 4323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54015cbd84da4e0aa48d8d447ed1c082. Latest consensus state: current_term: 1 leader_uuid: "54015cbd84da4e0aa48d8d447ed1c082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } } }
I20260923 18:09:58.654385 4323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:58.654763 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54015cbd84da4e0aa48d8d447ed1c082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54015cbd84da4e0aa48d8d447ed1c082" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 41877 } } }
I20260923 18:09:58.654938 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082 [sys.catalog]: This master's current role is: LEADER
I20260923 18:09:58.655658 4328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:09:58.659096 4328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:09:58.665912 4328 catalog_manager.cc:1347] Generated new cluster ID: 5516a0d332dc40ff908c7a113f39a9cb
I20260923 18:09:58.666021 4328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:09:58.684404 4328 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:09:58.685894 4328 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:09:58.698544 4328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: Generated new TSK 0
I20260923 18:09:58.699589 4328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:09:58.737563 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:58.861886 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:58.862210 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:58.862286 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:58.866626 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:58.866716 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:58.866802 4342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:09:58.873103 4342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:58.874591 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:58.875746 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:58.886919 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:58.886974 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:58.886919 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:58.887419 4342 server_base.cc:1034] running on GCE node
I20260923 18:09:58.888008 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:58.888715 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:58.889925 4342 hybrid_clock.cc:648] HybridClock initialized: now 1790186998889901 us; error 37 us; skew 500 ppm
Sep 23 18:09:58 dist-test-slave-rvjj krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:58.892956 4342 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:09:58.893883 4342 webserver.cc:460] Webserver started at http://127.18.44.193:33363/ using document root <none> and password file <none>
I20260923 18:09:58.894426 4342 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:58.894486 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:58.894658 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:58.896814 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "b7b510a89d124ca7801aa9276cfca547"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "b148f0bc1a5ae6231d4978505c45e44c"
server_key_iv: "28323d92c6ebf94b72b1c54af0e3a408"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.897367 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "b7b510a89d124ca7801aa9276cfca547"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "b148f0bc1a5ae6231d4978505c45e44c"
server_key_iv: "28323d92c6ebf94b72b1c54af0e3a408"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.901264 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:09:58.903920 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:58.905407 4342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:09:58.905529 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "b7b510a89d124ca7801aa9276cfca547"
format_stamp: "Formatted at 2026-09-23 18:09:58 on dist-test-slave-rvjj"
server_key: "b148f0bc1a5ae6231d4978505c45e44c"
server_key_iv: "28323d92c6ebf94b72b1c54af0e3a408"
server_key_version: "encryptionkey@0"
I20260923 18:09:58.905650 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:58.937487 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:58.941205 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:58.941395 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:58.942096 4342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:58.943053 4342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:58.943109 4342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:58.943153 4342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:58.943176 4342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:58.952901 4342 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:33669
I20260923 18:09:58.953024 4470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:33669 every 8 connection(s)
I20260923 18:09:58.953987 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:09:58.956434 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4342
I20260923 18:09:58.956568 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:09:58.956869 18611 external_mini_cluster.cc:1428] Setting key 9b62da963070cc093763527a766fce66
Sep 23 18:09:58 dist-test-slave-rvjj krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186998, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:09:58.971369 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:58.956120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:44985 (local address 127.18.44.254:41877)
0923 18:09:58.956269 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:58.956273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:58.957287 (+ 1014us) server_negotiation.cc:407] Connection header received
0923 18:09:58.958789 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:58.958792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:58.958850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:58.958921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:58.960580 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.961160 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.962988 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.963316 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.963428 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:58.966748 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:58.966752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:58.966754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:58.966785 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:58.968440 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:58.969132 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:58.969135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:58.969137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:58.969233 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:58.969772 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:58.969776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:58.969778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:58.970111 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:58.970224 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:58.971020 (+ 796us) server_negotiation.cc:299] Negotiation successful
0923 18:09:58.971159 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:09:58.972702 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:58.956616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41877 (local address 127.18.44.193:44985)
0923 18:09:58.957111 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:58.957171 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:58.958539 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:58.959126 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:58.959134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:58.959620 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:58.960354 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:58.960379 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.961319 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.961323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:58.962800 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:58.962815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:58.963524 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:58.963530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:58.963770 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:58.964486 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:58.964509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:58.966468 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:58.968627 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:58.968634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:58.968651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:58.968996 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:58.969398 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:58.969400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:58.969401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:58.969658 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:58.970257 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:58.970266 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:58.970670 (+ 404us) client_negotiation.cc:769] Sending connection context
0923 18:09:58.970973 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 18:09:58.971442 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260923 18:09:58.974516 4471 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41877
I20260923 18:09:58.974856 4471 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:58.975625 4471 heartbeater.cc:511] Master 127.18.44.254:41877 requested a full tablet report, sending...
I20260923 18:09:58.977727 4280 ts_manager.cc:194] Registered new tserver with Master: b7b510a89d124ca7801aa9276cfca547 (127.18.44.193:33669)
I20260923 18:09:58.979328 4280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:44985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:09:58.993229 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:59.112380 4476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:59.112695 4476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:59.112741 4476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:59.117175 4476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:59.117261 4476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:59.117326 4476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:09:59.123167 4476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:59.124387 4476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:59.125465 4476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:59.136893 4481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:59.136883 4482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:59.136942 4476 server_base.cc:1034] running on GCE node
W20260923 18:09:59.136859 4484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:59.137627 4476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:59.138375 4476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:59.139539 4476 hybrid_clock.cc:648] HybridClock initialized: now 1790186999139518 us; error 36 us; skew 500 ppm
Sep 23 18:09:59 dist-test-slave-rvjj krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:59.143018 4476 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:09:59.144405 4476 webserver.cc:460] Webserver started at http://127.18.44.194:39607/ using document root <none> and password file <none>
I20260923 18:09:59.145035 4476 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:59.145113 4476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:59.145330 4476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:59.148088 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "723543ea73d2472d8cf469333bc7ba89"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "c7cd62325a66f064e57c6373bd5a0066"
server_key_iv: "a87f56ef02cc3b0a72032b98dc2eb595"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.148900 4476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "723543ea73d2472d8cf469333bc7ba89"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "c7cd62325a66f064e57c6373bd5a0066"
server_key_iv: "a87f56ef02cc3b0a72032b98dc2eb595"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.156847 4476 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260923 18:09:59.160264 4491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.161576 4476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:09:59.161715 4476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "723543ea73d2472d8cf469333bc7ba89"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "c7cd62325a66f064e57c6373bd5a0066"
server_key_iv: "a87f56ef02cc3b0a72032b98dc2eb595"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.161840 4476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:59.176203 4476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:59.179893 4476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:59.180112 4476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:59.180826 4476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:59.181753 4476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:59.181806 4476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.181864 4476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:59.181893 4476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.191140 4476 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:45441
I20260923 18:09:59.191224 4604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:45441 every 8 connection(s)
I20260923 18:09:59.192191 4476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:09:59.195967 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4476
I20260923 18:09:59.196131 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:09:59.196648 18611 external_mini_cluster.cc:1428] Setting key ede74818704cda4ecf56495997702a4c
Sep 23 18:09:59 dist-test-slave-rvjj krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:09:59.210834 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.194245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:34141 (local address 127.18.44.254:41877)
0923 18:09:59.194381 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.194385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.195615 (+ 1230us) server_negotiation.cc:407] Connection header received
0923 18:09:59.197621 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.197720 (+ 99us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.197791 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.197883 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:59.199715 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.200341 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.202138 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.202569 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.202685 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.206006 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:59.206011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:59.206014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:59.206051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:59.207906 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.208607 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.208610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.208612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.208704 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.209276 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.209279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.209280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.209658 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:59.209745 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.210507 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.210661 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:09:59.211736 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.194725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41877 (local address 127.18.44.194:34141)
0923 18:09:59.195401 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.195459 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.197326 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.198107 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.198116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.198684 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:59.199502 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.199529 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.200536 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.200540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.201962 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.201975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.202782 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.202786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.202974 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.203825 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:59.203844 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:59.205763 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:59.208070 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:59.208078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:59.208096 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:59.208482 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:59.208865 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:59.208868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:59.208870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:59.209125 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:59.209820 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:59.209827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:59.210119 (+ 292us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.210343 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.210742 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":139,"threads_started":1}
I20260923 18:09:59.213148 4605 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41877
I20260923 18:09:59.213418 4605 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:59.213990 4605 heartbeater.cc:511] Master 127.18.44.254:41877 requested a full tablet report, sending...
I20260923 18:09:59.215145 4280 ts_manager.cc:194] Registered new tserver with Master: 723543ea73d2472d8cf469333bc7ba89 (127.18.44.194:45441)
I20260923 18:09:59.216010 4280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:34141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:09:59.230654 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:09:59.346319 4610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:59.346698 4610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:59.346757 4610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:59.351039 4610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:59.351135 4610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:59.351198 4610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:09:59.356979 4610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:59.358065 4610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:59.359067 4610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:09:59.369222 4610 server_base.cc:1034] running on GCE node
W20260923 18:09:59.369179 4618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:59.369153 4615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:59.369158 4616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:59.369870 4610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:59.370623 4610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:59.371788 4610 hybrid_clock.cc:648] HybridClock initialized: now 1790186999371766 us; error 51 us; skew 500 ppm
Sep 23 18:09:59 dist-test-slave-rvjj krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:09:59.374835 4610 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:09:59.375813 4610 webserver.cc:460] Webserver started at http://127.18.44.195:35403/ using document root <none> and password file <none>
I20260923 18:09:59.376451 4610 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:59.376530 4610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:59.376744 4610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:59.378872 4610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "8568144d12094a93bce6dfb5b0c31f39"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "281232fd69ba3e7fc6281cf8ec99701c"
server_key_iv: "f637f66e59a725418fd9bf0b540e067b"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.379520 4610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "8568144d12094a93bce6dfb5b0c31f39"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "281232fd69ba3e7fc6281cf8ec99701c"
server_key_iv: "f637f66e59a725418fd9bf0b540e067b"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.383608 4610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:09:59.386179 4625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.387778 4610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:09:59.387912 4610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "8568144d12094a93bce6dfb5b0c31f39"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "281232fd69ba3e7fc6281cf8ec99701c"
server_key_iv: "f637f66e59a725418fd9bf0b540e067b"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.388025 4610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:59.403019 4610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:59.406968 4610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:59.407176 4610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:09:59.407847 4610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:09:59.408804 4610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:09:59.408854 4610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.408911 4610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:09:59.408960 4610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 18:09:59.418946 4610 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:33909
I20260923 18:09:59.419003 4738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:33909 every 8 connection(s)
I20260923 18:09:59.420020 4610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:09:59.428228 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4610
I20260923 18:09:59.428354 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:09:59.428717 18611 external_mini_cluster.cc:1428] Setting key 023818d743901455ec0236d2c6b35a36
Sep 23 18:09:59 dist-test-slave-rvjj krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790186999, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:09:59.435320 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.422114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:47837 (local address 127.18.44.254:41877)
0923 18:09:59.422270 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.422274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.423271 (+ 997us) server_negotiation.cc:407] Connection header received
0923 18:09:59.424707 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.424711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.424782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.424857 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:59.426270 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.426812 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.428139 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.428387 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.428503 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.431478 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:59.431482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:59.431484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:59.431516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:59.433049 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.433654 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.433658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.433659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.433742 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.434108 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.434111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.434113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.434315 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:59.434402 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.434969 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.435106 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:09:59.436148 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.422520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:41877 (local address 127.18.44.195:47837)
0923 18:09:59.423077 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.423125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.424433 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.425096 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.425103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.425500 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:09:59.426083 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.426098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.426957 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.426960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.427999 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.428011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.428571 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.428576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.428802 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.429538 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:09:59.429561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:09:59.431308 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:09:59.433200 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:59.433207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:59.433226 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:59.433536 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:59.433833 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:09:59.433836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:09:59.433838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:09:59.433997 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:09:59.434420 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:09:59.434427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:09:59.434708 (+ 281us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.434892 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.435287 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":124,"threads_started":1}
I20260923 18:09:59.437377 4739 heartbeater.cc:347] Connected to a master server at 127.18.44.254:41877
I20260923 18:09:59.437630 4739 heartbeater.cc:464] Registering TS with master...
I20260923 18:09:59.438162 4739 heartbeater.cc:511] Master 127.18.44.254:41877 requested a full tablet report, sending...
I20260923 18:09:59.439536 4280 ts_manager.cc:194] Registered new tserver with Master: 8568144d12094a93bce6dfb5b0c31f39 (127.18.44.195:33909)
I20260923 18:09:59.440130 4280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:47837
I20260923 18:09:59.443524 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:09:59.454293 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.445842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.18.44.254:41877)
0923 18:09:59.445997 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.446002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.446100 (+ 98us) server_negotiation.cc:407] Connection header received
0923 18:09:59.446326 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.446329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.446375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.446446 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:09:59.447192 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.447798 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.448648 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.448873 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.448966 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.450818 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:09:59.450821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:09:59.450823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:09:59.450854 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:09:59.452315 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.452905 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.452908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.452909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.452992 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:09:59.453306 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:09:59.453309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:09:59.453310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:09:59.453541 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:09:59.453684 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.454044 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.454142 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:09:59.457882 4280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:09:59.461066 4280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:09:59.476832 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.470552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.18.44.195:33909)
0923 18:09:59.471753 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.471764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.471970 (+ 206us) server_negotiation.cc:407] Connection header received
0923 18:09:59.472136 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.472141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.472325 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.472492 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:59.472836 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.473680 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.475246 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476106 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476250 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476421 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.476556 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.476666 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":154,"threads_started":1}
I20260923 18:09:59.476825 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.470347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:33909 (local address 127.0.0.1:47394)
0923 18:09:59.471894 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.471907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.471989 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.472623 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.472626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.472644 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:59.472722 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.472727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.473823 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.473827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.475101 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.475112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476249 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.476359 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476398 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.476510 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.476647 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":1145,"thread_start_us":107,"threads_started":1}
I20260923 18:09:59.476855 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.470347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33669 (local address 127.0.0.1:55512)
0923 18:09:59.471300 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.471360 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.471500 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.472275 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.472281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.472304 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:59.472477 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.472486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.474410 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.474413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.475535 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.475542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476533 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.476625 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476646 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.476707 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.476765 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":104,"threads_started":1}
I20260923 18:09:59.477180 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.471063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.18.44.193:33669)
0923 18:09:59.471791 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.471803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.471825 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:09:59.471903 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.471908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.472116 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.472293 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:59.472598 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.474283 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.475641 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476189 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476473 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.476924 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.477028 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":151,"threads_started":1}
I20260923 18:09:59.477296 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.470552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:45441 (local address 127.0.0.1:34116)
0923 18:09:59.471780 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.471795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.471903 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.472463 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.472466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.472483 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:59.472564 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.472569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.473941 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.473944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.475385 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.475396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476636 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.476765 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476791 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.476872 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.477156 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":36,"thread_start_us":100,"threads_started":1}
I20260923 18:09:59.478044 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.471123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.18.44.194:45441)
0923 18:09:59.471613 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.471627 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.471896 (+ 269us) server_negotiation.cc:407] Connection header received
0923 18:09:59.472051 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.472055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.472203 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.472324 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:59.472692 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.473802 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.475528 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.476329 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.476438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.476570 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.477574 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.477891 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":166,"threads_started":1}
I20260923 18:09:59.480466 4539 tablet_service.cc:1467] Processing CreateTablet for tablet 514e572bba4f4232b883a60a0a624294 (DEFAULT_TABLE table=test-table [id=ed91ee0ccf5e4f4983279fb5a15bbeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:59.480552 4405 tablet_service.cc:1467] Processing CreateTablet for tablet 514e572bba4f4232b883a60a0a624294 (DEFAULT_TABLE table=test-table [id=ed91ee0ccf5e4f4983279fb5a15bbeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:59.480553 4673 tablet_service.cc:1467] Processing CreateTablet for tablet 514e572bba4f4232b883a60a0a624294 (DEFAULT_TABLE table=test-table [id=ed91ee0ccf5e4f4983279fb5a15bbeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:09:59.481577 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514e572bba4f4232b883a60a0a624294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:59.481602 4673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514e572bba4f4232b883a60a0a624294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:59.481602 4405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514e572bba4f4232b883a60a0a624294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:09:59.488422 4756 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Bootstrap starting.
I20260923 18:09:59.489609 4757 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: Bootstrap starting.
I20260923 18:09:59.491451 4756 tablet_bootstrap.cc:654] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:59.491595 4757 tablet_bootstrap.cc:654] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:59.491701 4758 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Bootstrap starting.
I20260923 18:09:59.492419 4757 log.cc:824] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:59.492802 4756 log.cc:824] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:59.493917 4758 tablet_bootstrap.cc:654] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Neither blocks nor log segments found. Creating new log.
I20260923 18:09:59.494426 4757 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: No bootstrap required, opened a new log
I20260923 18:09:59.494625 4757 ts_tablet_manager.cc:1397] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:09:59.495870 4756 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: No bootstrap required, opened a new log
I20260923 18:09:59.495224 4758 log.cc:824] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Log is configured to *not* fsync() on all Append() calls
I20260923 18:09:59.496093 4756 ts_tablet_manager.cc:1397] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260923 18:09:59.498162 4757 raft_consensus.cc:348] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.498420 4757 raft_consensus.cc:374] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:59.498493 4757 raft_consensus.cc:729] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b510a89d124ca7801aa9276cfca547, State: Initialized, Role: FOLLOWER
I20260923 18:09:59.499011 4757 consensus_queue.cc:260] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.499130 4756 raft_consensus.cc:348] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.499433 4756 raft_consensus.cc:374] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:59.499503 4756 raft_consensus.cc:729] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723543ea73d2472d8cf469333bc7ba89, State: Initialized, Role: FOLLOWER
I20260923 18:09:59.499858 4471 heartbeater.cc:503] Master 127.18.44.254:41877 was elected leader, sending a full tablet report...
I20260923 18:09:59.499975 4756 consensus_queue.cc:260] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.500130 4757 ts_tablet_manager.cc:1428] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:09:59.500635 4758 tablet_bootstrap.cc:492] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: No bootstrap required, opened a new log
I20260923 18:09:59.500665 4605 heartbeater.cc:503] Master 127.18.44.254:41877 was elected leader, sending a full tablet report...
I20260923 18:09:59.500837 4758 ts_tablet_manager.cc:1397] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260923 18:09:59.500862 4756 ts_tablet_manager.cc:1428] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:09:59.504599 4758 raft_consensus.cc:348] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.504958 4758 raft_consensus.cc:374] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:09:59.505084 4758 raft_consensus.cc:729] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8568144d12094a93bce6dfb5b0c31f39, State: Initialized, Role: FOLLOWER
I20260923 18:09:59.505611 4758 consensus_queue.cc:260] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.506422 4739 heartbeater.cc:503] Master 127.18.44.254:41877 was elected leader, sending a full tablet report...
I20260923 18:09:59.506609 4758 ts_tablet_manager.cc:1428] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:09:59.630280 4764 raft_consensus.cc:482] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:09:59.630584 4764 raft_consensus.cc:504] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.632138 4764 leader_election.cc:288] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7b510a89d124ca7801aa9276cfca547 (127.18.44.193:33669), 723543ea73d2472d8cf469333bc7ba89 (127.18.44.194:45441)
I20260923 18:09:59.637207 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.633057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:38437 (local address 127.18.44.194:45441)
0923 18:09:59.633335 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.633341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.633356 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:09:59.633409 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.633411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.633456 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.633525 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:59.633834 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.634541 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.635834 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.636428 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.636576 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.636647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.636913 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.637059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260923 18:09:59.637331 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.632638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:40513 (local address 127.18.44.193:33669)
0923 18:09:59.632771 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.632776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.632793 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:09:59.632956 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.632959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.633029 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.633112 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:09:59.633560 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.634379 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.636093 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.636691 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.636812 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.636867 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.637082 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.637182 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:09:59.637336 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.632421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33669 (local address 127.18.44.195:40513)
0923 18:09:59.632674 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.632692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.632805 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.633182 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.633187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.633226 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:59.633379 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.633387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.634524 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.634527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.635919 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.635932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.636820 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.636824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.636949 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.636970 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.637050 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.637156 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1}
I20260923 18:09:59.637733 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.632567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:45441 (local address 127.18.44.195:38437)
0923 18:09:59.633132 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.633147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.633244 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.633569 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:09:59.633572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:09:59.633589 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:09:59.633708 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.633714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.634675 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.634679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.635692 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:09:59.635704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.636581 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.636585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:09:59.636698 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.636728 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 18:09:59.637426 (+ 698us) client_negotiation.cc:240] Negotiation successful
0923 18:09:59.637541 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":40,"thread_start_us":151,"threads_started":1}
I20260923 18:09:59.638338 4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514e572bba4f4232b883a60a0a624294" candidate_uuid: "8568144d12094a93bce6dfb5b0c31f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b510a89d124ca7801aa9276cfca547" is_pre_election: true
I20260923 18:09:59.638489 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514e572bba4f4232b883a60a0a624294" candidate_uuid: "8568144d12094a93bce6dfb5b0c31f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723543ea73d2472d8cf469333bc7ba89" is_pre_election: true
I20260923 18:09:59.638778 4425 raft_consensus.cc:2454] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8568144d12094a93bce6dfb5b0c31f39 in term 0.
I20260923 18:09:59.638846 4559 raft_consensus.cc:2454] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8568144d12094a93bce6dfb5b0c31f39 in term 0.
I20260923 18:09:59.639456 4629 leader_election.cc:302] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723543ea73d2472d8cf469333bc7ba89, 8568144d12094a93bce6dfb5b0c31f39; no voters:
I20260923 18:09:59.639830 4764 raft_consensus.cc:2781] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:09:59.639936 4764 raft_consensus.cc:482] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:09:59.639987 4764 raft_consensus.cc:3037] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:59.641316 4764 raft_consensus.cc:504] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.641780 4764 leader_election.cc:288] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [CANDIDATE]: Term 1 election: Requested vote from peers b7b510a89d124ca7801aa9276cfca547 (127.18.44.193:33669), 723543ea73d2472d8cf469333bc7ba89 (127.18.44.194:45441)
I20260923 18:09:59.642254 4425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514e572bba4f4232b883a60a0a624294" candidate_uuid: "8568144d12094a93bce6dfb5b0c31f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b510a89d124ca7801aa9276cfca547"
I20260923 18:09:59.642323 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514e572bba4f4232b883a60a0a624294" candidate_uuid: "8568144d12094a93bce6dfb5b0c31f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723543ea73d2472d8cf469333bc7ba89"
I20260923 18:09:59.642419 4425 raft_consensus.cc:3037] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:59.642493 4559 raft_consensus.cc:3037] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:09:59.643862 4425 raft_consensus.cc:2454] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8568144d12094a93bce6dfb5b0c31f39 in term 1.
I20260923 18:09:59.643886 4559 raft_consensus.cc:2454] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8568144d12094a93bce6dfb5b0c31f39 in term 1.
I20260923 18:09:59.644276 4629 leader_election.cc:302] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723543ea73d2472d8cf469333bc7ba89, 8568144d12094a93bce6dfb5b0c31f39; no voters:
I20260923 18:09:59.644502 4764 raft_consensus.cc:2781] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:09:59.644829 4764 raft_consensus.cc:686] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 1 LEADER]: Becoming Leader. State: Replica: 8568144d12094a93bce6dfb5b0c31f39, State: Running, Role: LEADER
I20260923 18:09:59.645282 4764 consensus_queue.cc:237] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } }
I20260923 18:09:59.649538 4280 catalog_manager.cc:5495] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8568144d12094a93bce6dfb5b0c31f39 (127.18.44.195). New cstate: current_term: 1 leader_uuid: "8568144d12094a93bce6dfb5b0c31f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8568144d12094a93bce6dfb5b0c31f39" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 33909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 } health_report { overall_health: UNKNOWN } } }
W20260923 18:09:59.671056 4740 tablet.cc:2369] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:59.678359 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:09:59.674492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.18.44.195:33909)
0923 18:09:59.674680 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 18:09:59.674685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:09:59.674699 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:09:59.674777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:09:59.674779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:09:59.674846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:09:59.674945 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:09:59.675237 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.675893 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.676815 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:09:59.677022 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:09:59.677108 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:09:59.677444 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:09:59.677867 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:09:59.678000 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 18:09:59.678133 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 18:09:59.678181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 18:09:59.686672 4559 raft_consensus.cc:1261] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 1 FOLLOWER]: Refusing update from remote peer 8568144d12094a93bce6dfb5b0c31f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:59.686741 4425 raft_consensus.cc:1261] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 1 FOLLOWER]: Refusing update from remote peer 8568144d12094a93bce6dfb5b0c31f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:09:59.687592 4764 consensus_queue.cc:1035] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723543ea73d2472d8cf469333bc7ba89" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:09:59.687875 4766 consensus_queue.cc:1035] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b510a89d124ca7801aa9276cfca547" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 18:09:59.693307 4606 tablet.cc:2369] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:09:59.697328 4769 mvcc.cc:205] Tried to move back new op lower bound from 7332605950706483200 to 7332605950550405120. Current Snapshot: MvccSnapshot[applied={T|T < 7332605950706483200}]
I20260923 18:09:59.698148 4770 mvcc.cc:205] Tried to move back new op lower bound from 7332605950706483200 to 7332605950550405120. Current Snapshot: MvccSnapshot[applied={T|T < 7332605950706483200}]
I20260923 18:09:59.714118 4278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:59.714448 4278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:09:59.718196 4278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54015cbd84da4e0aa48d8d447ed1c082: Sending DeleteTablet for 3 replicas of tablet 514e572bba4f4232b883a60a0a624294
I20260923 18:09:59.719434 4673 tablet_service.cc:1514] Processing DeleteTablet for tablet 514e572bba4f4232b883a60a0a624294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:59 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:47394
I20260923 18:09:59.719434 4539 tablet_service.cc:1514] Processing DeleteTablet for tablet 514e572bba4f4232b883a60a0a624294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:59 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:34116
I20260923 18:09:59.719486 4405 tablet_service.cc:1514] Processing DeleteTablet for tablet 514e572bba4f4232b883a60a0a624294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:09:59 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 18:09:59.720118 4779 tablet_replica.cc:330] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: stopping tablet replica
I20260923 18:09:59.720327 4780 tablet_replica.cc:330] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547: stopping tablet replica
I20260923 18:09:59.720638 4779 raft_consensus.cc:2229] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:09:59.720638 4780 raft_consensus.cc:2229] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:59.721104 4780 raft_consensus.cc:2258] T 514e572bba4f4232b883a60a0a624294 P b7b510a89d124ca7801aa9276cfca547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:59.721167 4779 raft_consensus.cc:2258] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:59.721580 4778 tablet_replica.cc:330] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: stopping tablet replica
I20260923 18:09:59.721613 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4342
I20260923 18:09:59.721940 4778 raft_consensus.cc:2229] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:09:59.722293 4778 raft_consensus.cc:2258] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:09:59.724352 4779 ts_tablet_manager.cc:1905] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:59.726902 4778 ts_tablet_manager.cc:1905] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:09:59.728359 4779 ts_tablet_manager.cc:1918] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:59.728525 4779 log.cc:1196] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/514e572bba4f4232b883a60a0a624294
I20260923 18:09:59.729132 4779 ts_tablet_manager.cc:1939] T 514e572bba4f4232b883a60a0a624294 P 8568144d12094a93bce6dfb5b0c31f39: Deleting consensus metadata
I20260923 18:09:59.731076 4778 ts_tablet_manager.cc:1918] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:09:59.731221 4778 log.cc:1196] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/514e572bba4f4232b883a60a0a624294
I20260923 18:09:59.731302 4265 catalog_manager.cc:4841] TS 8568144d12094a93bce6dfb5b0c31f39 (127.18.44.195:33909): tablet 514e572bba4f4232b883a60a0a624294 (table test-table [id=ed91ee0ccf5e4f4983279fb5a15bbeb1]) successfully deleted
I20260923 18:09:59.731629 4778 ts_tablet_manager.cc:1939] T 514e572bba4f4232b883a60a0a624294 P 723543ea73d2472d8cf469333bc7ba89: Deleting consensus metadata
W20260923 18:09:59.731897 4264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:33669 (error 108)
I20260923 18:09:59.733563 4267 catalog_manager.cc:4841] TS 723543ea73d2472d8cf469333bc7ba89 (127.18.44.194:45441): tablet 514e572bba4f4232b883a60a0a624294 (table test-table [id=ed91ee0ccf5e4f4983279fb5a15bbeb1]) successfully deleted
W20260923 18:09:59.733584 4755 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:09:59.732957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:33669 (local address 127.0.0.1:55518)
0923 18:09:59.733144 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 18:09:59.733168 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 18:09:59.733319 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:09:59.733439 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33669: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 18:09:59.733990 4264 catalog_manager.cc:4584] TS b7b510a89d124ca7801aa9276cfca547 (127.18.44.193:33669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 514e572bba4f4232b883a60a0a624294: Network error: Client connection negotiation failed: client connection to 127.18.44.193:33669: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260923 18:09:59.734153 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4476
I20260923 18:09:59.741775 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4610
I20260923 18:09:59.748296 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4248
2026-09-23T18:09:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2454 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T18:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:09:59Z Disabled control of system clock
I20260923 18:09:59.784047 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36317
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:09:59.931764 4785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:09:59.932224 4785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:09:59.932271 4785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:09:59.937129 4785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:09:59.937256 4785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:09:59.937273 4785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:09:59.937291 4785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:09:59.937306 4785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:09:59.943544 4785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:36317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:09:59.945035 4785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:09:59.946786 4785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:09:59.957338 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:59.957394 4790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:09:59.957343 4793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:09:59.957653 4785 server_base.cc:1034] running on GCE node
I20260923 18:09:59.958498 4785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:09:59.960114 4785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:09:59.961306 4785 hybrid_clock.cc:648] HybridClock initialized: now 1790186999961274 us; error 54 us; skew 500 ppm
I20260923 18:09:59.963935 4785 webserver.cc:460] Webserver started at http://127.18.44.254:46363/ using document root <none> and password file <none>
I20260923 18:09:59.964780 4785 fs_manager.cc:359] Metadata directory not provided
I20260923 18:09:59.964864 4785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:09:59.965128 4785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:09:59.967886 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "ab31dcf8863641bbb3f6737e4ce1271f"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "6b4fa1e44526bfc36b338df5a912b98e"
server_key_iv: "1f77bf2eb766bd7c59d4b034bd60142a"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.968600 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "ab31dcf8863641bbb3f6737e4ce1271f"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "6b4fa1e44526bfc36b338df5a912b98e"
server_key_iv: "1f77bf2eb766bd7c59d4b034bd60142a"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.973727 4785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:09:59.977455 4799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:09:59.979471 4785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:09:59.979677 4785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "ab31dcf8863641bbb3f6737e4ce1271f"
format_stamp: "Formatted at 2026-09-23 18:09:59 on dist-test-slave-rvjj"
server_key: "6b4fa1e44526bfc36b338df5a912b98e"
server_key_iv: "1f77bf2eb766bd7c59d4b034bd60142a"
server_key_version: "encryptionkey@0"
I20260923 18:09:59.979835 4785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:09:59.993191 4785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:09:59.994225 4785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:09:59.994467 4785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:00.010313 4785 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:36317
I20260923 18:10:00.010316 4851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:36317 every 8 connection(s)
I20260923 18:10:00.013769 4785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:00.018131 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:00.024251 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4785
I20260923 18:10:00.024444 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:00.024731 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: Bootstrap starting.
I20260923 18:10:00.024881 18611 external_mini_cluster.cc:1428] Setting key 41658bce6f0c95e94119a7df833893a4
I20260923 18:10:00.028195 4852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:00.029287 4852 log.cc:824] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:00.032163 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: No bootstrap required, opened a new log
I20260923 18:10:00.037439 4852 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } }
I20260923 18:10:00.037752 4852 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:00.037892 4852 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab31dcf8863641bbb3f6737e4ce1271f, State: Initialized, Role: FOLLOWER
I20260923 18:10:00.038503 4852 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } }
I20260923 18:10:00.038674 4852 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:00.038748 4852 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:00.038868 4852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:00.040006 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.027866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.18.44.254:36317)
0923 18:10:00.029215 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.029243 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.029308 (+ 65us) server_negotiation.cc:407] Connection header received
0923 18:10:00.030317 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.030343 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.030857 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.031360 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:00.031870 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.033077 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.034627 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.035469 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.035886 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.037131 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:00.037138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:00.037162 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:00.037752 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:00.037878 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:00.037915 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:00.038083 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.038316 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.038767 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":153,"threads_started":1}
I20260923 18:10:00.040019 4852 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } }
I20260923 18:10:00.040621 4852 leader_election.cc:302] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab31dcf8863641bbb3f6737e4ce1271f; no voters:
I20260923 18:10:00.041056 4852 leader_election.cc:288] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:00.041167 4857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:00.041354 4857 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [term 1 LEADER]: Becoming Leader. State: Replica: ab31dcf8863641bbb3f6737e4ce1271f, State: Running, Role: LEADER
I20260923 18:10:00.041719 4857 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } }
I20260923 18:10:00.043589 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } } }
I20260923 18:10:00.043795 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:00.043937 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab31dcf8863641bbb3f6737e4ce1271f. Latest consensus state: current_term: 1 leader_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab31dcf8863641bbb3f6737e4ce1271f" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 36317 } } }
I20260923 18:10:00.044039 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:00.044627 4852 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:00.044901 4861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:00.049130 4861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:00.057384 4861 catalog_manager.cc:1347] Generated new cluster ID: a0703aa07ad04419aa37b4bdfb72a7e6
I20260923 18:10:00.057507 4861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:00.075325 4861 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:00.076713 4861 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:00.087867 4861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: Generated new TSK 0
I20260923 18:10:00.088922 4861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:00.099436 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--builtin_ntp_servers=127.18.44.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:10:00.248684 4876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:00.249094 4876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:00.249161 4876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:00.253966 4876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:00.254094 4876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:00.254190 4876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:10:00.260432 4876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:00.262019 4876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:00.263955 4876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:00.277925 4884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:00.277925 4882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:00.278017 4876 server_base.cc:1034] running on GCE node
W20260923 18:10:00.277925 4881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:00.278925 4876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:00.279929 4876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:00.281294 4876 hybrid_clock.cc:648] HybridClock initialized: now 1790187000281247 us; error 64 us; skew 500 ppm
I20260923 18:10:00.297590 4876 webserver.cc:460] Webserver started at http://127.18.44.193:37229/ using document root <none> and password file <none>
I20260923 18:10:00.298930 4876 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:00.299103 4876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:00.299504 4876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:00.303658 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "3af918236ad840619f3d13b000e40e5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "3f29d98af0e410c48e327a6fcab59ef5"
server_key_iv: "e457541b898bbebb5babf46e0d03d1c1"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.304616 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "3af918236ad840619f3d13b000e40e5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "3f29d98af0e410c48e327a6fcab59ef5"
server_key_iv: "e457541b898bbebb5babf46e0d03d1c1"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.311676 4876 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 18:10:00.316735 4890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.319219 4876 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:10:00.319540 4876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "3af918236ad840619f3d13b000e40e5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "3f29d98af0e410c48e327a6fcab59ef5"
server_key_iv: "e457541b898bbebb5babf46e0d03d1c1"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.320166 4876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:00.353266 4876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:00.354230 4876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:00.354713 4876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:00.355934 4876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:00.357384 4876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:00.357452 4876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.357510 4876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:00.357540 4876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.370842 4876 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:41433
I20260923 18:10:00.370901 5003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:41433 every 8 connection(s)
I20260923 18:10:00.372247 4876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:00.380681 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 4876
I20260923 18:10:00.380878 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:00.381503 18611 external_mini_cluster.cc:1428] Setting key 1503f3a0dace3aeea4185045e09fb4df
I20260923 18:10:00.385025 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--builtin_ntp_servers=127.18.44.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:10:00.390285 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.374816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:40655 (local address 127.18.44.254:36317)
0923 18:10:00.375047 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.375051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.376613 (+ 1562us) server_negotiation.cc:407] Connection header received
0923 18:10:00.378461 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.378465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.379025 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.379179 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:00.381878 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.382730 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.386022 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.386366 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.386512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.389518 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:00.389522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:00.389525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:00.389544 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:00.389576 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:00.389588 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:00.389678 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.390026 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.390116 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 18:10:00.392068 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.375217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36317 (local address 127.18.44.193:40655)
0923 18:10:00.376329 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.376404 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.378090 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.379565 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.379578 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.379727 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:00.381494 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.381534 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.383032 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.383038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.385797 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.385826 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.386808 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.386817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.387591 (+ 774us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.388773 (+ 1182us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:00.388775 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:00.388858 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:00.389002 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:00.389751 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:00.389764 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:00.389774 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.390574 (+ 800us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.391030 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":147,"threads_started":1}
I20260923 18:10:00.394744 5004 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36317
I20260923 18:10:00.395146 5004 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:00.395933 5004 heartbeater.cc:511] Master 127.18.44.254:36317 requested a full tablet report, sending...
I20260923 18:10:00.399041 4816 ts_manager.cc:194] Registered new tserver with Master: 3af918236ad840619f3d13b000e40e5a (127.18.44.193:41433)
I20260923 18:10:00.401367 4816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.193:40655
W20260923 18:10:00.568337 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:00.568744 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:00.568794 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:00.577297 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:00.577448 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:00.577527 5007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:10:00.584592 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:00.585994 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:00.588377 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:00.603353 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:00.603353 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:00.603438 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:00.603781 5007 server_base.cc:1034] running on GCE node
I20260923 18:10:00.604338 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:00.605150 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:00.606551 5007 hybrid_clock.cc:648] HybridClock initialized: now 1790187000606474 us; error 102 us; skew 500 ppm
I20260923 18:10:00.610119 5007 webserver.cc:460] Webserver started at http://127.18.44.194:45589/ using document root <none> and password file <none>
I20260923 18:10:00.610984 5007 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:00.611075 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:00.611377 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:00.613864 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "a882e6c3117d4f83b88df623ab870c5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "fa09a1589c64fc7c65fffb3b13f9dcd7"
server_key_iv: "374817d7d852fff8b98e9e844008358b"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.614717 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "a882e6c3117d4f83b88df623ab870c5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "fa09a1589c64fc7c65fffb3b13f9dcd7"
server_key_iv: "374817d7d852fff8b98e9e844008358b"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.620029 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 18:10:00.624600 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.627344 5007 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:10:00.627538 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "a882e6c3117d4f83b88df623ab870c5a"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "fa09a1589c64fc7c65fffb3b13f9dcd7"
server_key_iv: "374817d7d852fff8b98e9e844008358b"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.627683 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:00.657459 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:00.658301 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:00.658519 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:00.659226 5007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:00.660938 5007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:00.661335 5007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.661505 5007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:00.661675 5007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.674224 5007 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:39021
I20260923 18:10:00.674563 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:39021 every 8 connection(s)
I20260923 18:10:00.675431 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:00.679620 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5007
I20260923 18:10:00.679772 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:00.680092 18611 external_mini_cluster.cc:1428] Setting key d0238b72b64ed6564fd5d11139d3f6fd
I20260923 18:10:00.682902 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--builtin_ntp_servers=127.18.44.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:10:00.691529 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.679004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:39195 (local address 127.18.44.254:36317)
0923 18:10:00.679451 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.679457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.681521 (+ 2064us) server_negotiation.cc:407] Connection header received
0923 18:10:00.683762 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.683767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.683863 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.683951 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:00.686373 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.687103 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.688831 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.689094 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.689211 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.690831 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:00.690835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:00.690837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:00.690857 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:00.690884 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:00.690894 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:00.690963 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.691145 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.691311 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260923 18:10:00.693598 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.679486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36317 (local address 127.18.44.194:39195)
0923 18:10:00.681194 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.681290 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.683366 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.684277 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.684289 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.684548 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:00.685768 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.685798 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.687335 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.687338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.688605 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.688626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.689320 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.689325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.689564 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.690403 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:00.690405 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:00.690475 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:00.690590 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:00.690978 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:00.690989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:00.690997 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.691152 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.692046 (+ 894us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":195,"threads_started":1}
I20260923 18:10:00.695726 5135 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36317
I20260923 18:10:00.696465 5135 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:00.697853 5135 heartbeater.cc:511] Master 127.18.44.254:36317 requested a full tablet report, sending...
I20260923 18:10:00.700059 4816 ts_manager.cc:194] Registered new tserver with Master: a882e6c3117d4f83b88df623ab870c5a (127.18.44.194:39021)
I20260923 18:10:00.700941 4816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.194:39195
W20260923 18:10:00.822047 5138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:00.822377 5138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:00.822418 5138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:00.826959 5138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:00.827066 5138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:00.827128 5138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:10:00.833904 5138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:00.835371 5138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:00.837092 5138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:00.849422 5144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:00.849445 5143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:00.849440 5146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:00.850203 5138 server_base.cc:1034] running on GCE node
I20260923 18:10:00.850870 5138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:00.851752 5138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:00.853116 5138 hybrid_clock.cc:648] HybridClock initialized: now 1790187000853073 us; error 72 us; skew 500 ppm
I20260923 18:10:00.855885 5138 webserver.cc:460] Webserver started at http://127.18.44.195:39175/ using document root <none> and password file <none>
I20260923 18:10:00.856540 5138 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:00.856619 5138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:00.856841 5138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:00.859578 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "7f88d2c1c2b147d68f304af2b95ad053"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "a8c1369e02c1ecf995bbfa7d0472bffa"
server_key_iv: "bbf3404bfb0a4111eb3f2387a9ba55c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.860275 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "7f88d2c1c2b147d68f304af2b95ad053"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "a8c1369e02c1ecf995bbfa7d0472bffa"
server_key_iv: "bbf3404bfb0a4111eb3f2387a9ba55c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.866212 5138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 18:10:00.869719 5152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.872160 5138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:10:00.872341 5138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "7f88d2c1c2b147d68f304af2b95ad053"
format_stamp: "Formatted at 2026-09-23 18:10:00 on dist-test-slave-rvjj"
server_key: "a8c1369e02c1ecf995bbfa7d0472bffa"
server_key_iv: "bbf3404bfb0a4111eb3f2387a9ba55c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:00.872484 5138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:00.898152 5138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:00.899022 5138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:00.899237 5138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:00.899942 5138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:00.901371 5138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:00.901434 5138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.901495 5138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:00.901597 5138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:00.913825 5138 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:35217
I20260923 18:10:00.913848 5265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:35217 every 8 connection(s)
I20260923 18:10:00.915166 5138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:00.922899 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5138
I20260923 18:10:00.923075 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:00.923555 18611 external_mini_cluster.cc:1428] Setting key 82eb1cb428ebc6d3bf91d0572e5895d0
I20260923 18:10:00.932651 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.919056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:46381 (local address 127.18.44.254:36317)
0923 18:10:00.919472 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.919477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.921144 (+ 1667us) server_negotiation.cc:407] Connection header received
0923 18:10:00.923938 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.923945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.924076 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.924232 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:00.926383 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.927543 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.929526 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.929852 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.929993 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.931881 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:00.931887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:00.931890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:00.931920 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:00.931962 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:00.931975 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:00.932080 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.932366 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.932473 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260923 18:10:00.933701 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.919499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:36317 (local address 127.18.44.195:46381)
0923 18:10:00.920834 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.920931 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.923119 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.924683 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.924697 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.924971 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:00.926008 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.926038 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.927904 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.927908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.929118 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.929136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.930109 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.930113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.930354 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.931186 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:00.931188 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:00.931288 (+ 100us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:00.931525 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:00.932144 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:00.932154 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:00.932161 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.932304 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.932628 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":182,"threads_started":1}
I20260923 18:10:00.935971 5266 heartbeater.cc:347] Connected to a master server at 127.18.44.254:36317
I20260923 18:10:00.936313 5266 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:00.937016 5266 heartbeater.cc:511] Master 127.18.44.254:36317 requested a full tablet report, sending...
I20260923 18:10:00.938361 4815 ts_manager.cc:194] Registered new tserver with Master: 7f88d2c1c2b147d68f304af2b95ad053 (127.18.44.195:35217)
I20260923 18:10:00.939021 4815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.195:46381
I20260923 18:10:00.939695 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:00.949916 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.942194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.18.44.254:36317)
0923 18:10:00.942535 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.942540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.943120 (+ 580us) server_negotiation.cc:407] Connection header received
0923 18:10:00.943392 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.943395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.943456 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.943568 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:00.944097 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.944876 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.947132 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.947607 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.947756 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.949086 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:00.949090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:00.949092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:00.949110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:00.949139 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:00.949148 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:00.949329 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.949596 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.949651 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260923 18:10:00.956158 4815 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:00.960079 4815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:00.983692 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.975347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.18.44.194:39021)
0923 18:10:00.976450 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.976466 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.976492 (+ 26us) server_negotiation.cc:407] Connection header received
0923 18:10:00.976600 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.976607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.977051 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.977895 (+ 844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:00.978054 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.979328 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.981714 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.982534 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.982682 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.982925 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.983325 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.983467 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":301,"threads_started":1}
I20260923 18:10:00.983716 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.974466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39021 (local address 127.0.0.1:38110)
0923 18:10:00.975640 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.975736 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.975910 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.977621 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.977624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.977645 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:00.977738 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.977745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.979516 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.979520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.981527 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.981545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.982757 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.982760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.982931 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.983004 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.983225 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.983472 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":106,"thread_start_us":149,"threads_started":1}
I20260923 18:10:00.984512 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.974471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.18.44.193:41433)
0923 18:10:00.975971 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.975980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.976017 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:10:00.976404 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.976414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.976741 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.976975 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:00.977727 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.979327 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.981601 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.982534 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.982682 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.982925 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.983721 (+ 796us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.984192 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":150,"threads_started":1}
I20260923 18:10:00.985105 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.974400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41433 (local address 127.0.0.1:54442)
0923 18:10:00.975915 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.975935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.976157 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.977040 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.977047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.977186 (+ 139us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:00.977545 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.977557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.979547 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.979551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.981422 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.981437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.983240 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.983244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.983432 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.983459 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.984796 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.984930 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":254,"thread_start_us":120,"threads_started":1}
I20260923 18:10:00.986438 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.976705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:35217 (local address 127.0.0.1:33048)
0923 18:10:00.977789 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:00.977813 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:00.977966 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:00.978535 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:00.978539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:00.978558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:00.978699 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.978706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.980052 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.980057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.981527 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:00.981545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.985851 (+ 4306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.985884 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:00.986000 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.986032 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 18:10:00.986126 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:10:00.986238 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":8,"thread_start_us":139,"threads_started":1}
I20260923 18:10:00.988611 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:00.976273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.18.44.195:35217)
0923 18:10:00.976760 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:00.976778 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:00.977952 (+ 1174us) server_negotiation.cc:407] Connection header received
0923 18:10:00.978131 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:00.978137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:00.978372 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:00.978541 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:00.978842 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.979893 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.983049 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:00.985293 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:00.985442 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:00.985681 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:00.986380 (+ 699us) server_negotiation.cc:299] Negotiation successful
0923 18:10:00.987334 (+ 954us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":143,"threads_started":1}
I20260923 18:10:00.989113 4938 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8db61476dd4d7496340f564af2e6ab (DEFAULT_TABLE table=test-table [id=129d7cf28f804542bfe578eb1e5eb6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:00.990438 4938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8db61476dd4d7496340f564af2e6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:00.992166 5069 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8db61476dd4d7496340f564af2e6ab (DEFAULT_TABLE table=test-table [id=129d7cf28f804542bfe578eb1e5eb6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:00.993933 5069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8db61476dd4d7496340f564af2e6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:00.994072 5200 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8db61476dd4d7496340f564af2e6ab (DEFAULT_TABLE table=test-table [id=129d7cf28f804542bfe578eb1e5eb6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:00.996446 5200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8db61476dd4d7496340f564af2e6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:01.002259 5283 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: Bootstrap starting.
I20260923 18:10:01.006373 5284 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Bootstrap starting.
I20260923 18:10:01.007788 5283 tablet_bootstrap.cc:654] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:01.008715 5285 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Bootstrap starting.
I20260923 18:10:01.009150 5283 log.cc:824] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:01.011075 5284 tablet_bootstrap.cc:654] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:01.012392 5285 tablet_bootstrap.cc:654] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:01.012419 5284 log.cc:824] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:01.012871 5283 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: No bootstrap required, opened a new log
I20260923 18:10:01.013445 5283 ts_tablet_manager.cc:1397] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 18:10:01.013710 5285 log.cc:824] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:01.015497 5284 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: No bootstrap required, opened a new log
I20260923 18:10:01.015769 5284 ts_tablet_manager.cc:1397] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 18:10:01.017303 5285 tablet_bootstrap.cc:492] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: No bootstrap required, opened a new log
I20260923 18:10:01.017540 5285 ts_tablet_manager.cc:1397] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260923 18:10:01.023093 5284 raft_consensus.cc:348] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.023511 5284 raft_consensus.cc:374] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:01.023571 5284 raft_consensus.cc:729] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f88d2c1c2b147d68f304af2b95ad053, State: Initialized, Role: FOLLOWER
I20260923 18:10:01.023093 5285 raft_consensus.cc:348] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.023957 5285 raft_consensus.cc:374] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:01.024053 5285 raft_consensus.cc:729] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a882e6c3117d4f83b88df623ab870c5a, State: Initialized, Role: FOLLOWER
I20260923 18:10:01.024168 5284 consensus_queue.cc:260] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.024360 5283 raft_consensus.cc:348] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.024791 5283 raft_consensus.cc:374] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:01.024873 5283 raft_consensus.cc:729] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af918236ad840619f3d13b000e40e5a, State: Initialized, Role: FOLLOWER
I20260923 18:10:01.025076 5285 consensus_queue.cc:260] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.025465 5283 consensus_queue.cc:260] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.026103 5284 ts_tablet_manager.cc:1428] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260923 18:10:01.026569 5266 heartbeater.cc:503] Master 127.18.44.254:36317 was elected leader, sending a full tablet report...
I20260923 18:10:01.026598 5285 ts_tablet_manager.cc:1428] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260923 18:10:01.026829 5283 ts_tablet_manager.cc:1428] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260923 18:10:01.027243 5135 heartbeater.cc:503] Master 127.18.44.254:36317 was elected leader, sending a full tablet report...
I20260923 18:10:01.027638 5004 heartbeater.cc:503] Master 127.18.44.254:36317 was elected leader, sending a full tablet report...
I20260923 18:10:01.043179 5290 raft_consensus.cc:482] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:01.043484 5290 raft_consensus.cc:504] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.045079 5290 leader_election.cc:288] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af918236ad840619f3d13b000e40e5a (127.18.44.193:41433), 7f88d2c1c2b147d68f304af2b95ad053 (127.18.44.195:35217)
I20260923 18:10:01.050700 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.045643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:56951 (local address 127.18.44.193:41433)
0923 18:10:01.045829 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:01.045834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:01.045856 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:10:01.045920 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:01.045924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:01.045982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:01.046075 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:01.046489 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.047147 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.049076 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.049783 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.049941 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:01.050001 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:01.050485 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 18:10:01.050572 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 18:10:01.050738 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.045473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41433 (local address 127.18.44.194:56951)
0923 18:10:01.045642 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:01.045661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:01.045805 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:01.046170 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:01.046175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:01.046207 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:01.046341 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:01.046350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.047387 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.047390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:01.048832 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:01.048901 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.050086 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.050093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:01.050249 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:01.050287 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 18:10:01.050412 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 18:10:01.050525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260923 18:10:01.051618 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.046313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:34893 (local address 127.18.44.195:35217)
0923 18:10:01.046728 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:01.046732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:01.046752 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:01.046823 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:01.046826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:01.046881 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:01.046991 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:01.047375 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.048301 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.050095 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.050752 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.050895 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:01.050953 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:01.051379 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 18:10:01.051480 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260923 18:10:01.051679 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.045839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:35217 (local address 127.18.44.194:34893)
0923 18:10:01.046564 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:01.046582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:01.046677 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:01.047010 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:01.047014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:01.047032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:01.047164 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:01.047170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.048493 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.048497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:01.049827 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:01.049838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.051057 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.051060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:01.051173 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:01.051199 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:10:01.051352 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 18:10:01.051445 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":134,"thread_start_us":363,"threads_started":1}
I20260923 18:10:01.051659 4958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8db61476dd4d7496340f564af2e6ab" candidate_uuid: "a882e6c3117d4f83b88df623ab870c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af918236ad840619f3d13b000e40e5a" is_pre_election: true
I20260923 18:10:01.052026 4958 raft_consensus.cc:2454] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a882e6c3117d4f83b88df623ab870c5a in term 0.
I20260923 18:10:01.052392 5220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8db61476dd4d7496340f564af2e6ab" candidate_uuid: "a882e6c3117d4f83b88df623ab870c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f88d2c1c2b147d68f304af2b95ad053" is_pre_election: true
I20260923 18:10:01.052663 5022 leader_election.cc:302] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af918236ad840619f3d13b000e40e5a, a882e6c3117d4f83b88df623ab870c5a; no voters:
I20260923 18:10:01.052784 5220 raft_consensus.cc:2454] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a882e6c3117d4f83b88df623ab870c5a in term 0.
I20260923 18:10:01.053099 5290 raft_consensus.cc:2781] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:01.053218 5290 raft_consensus.cc:482] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:01.053283 5290 raft_consensus.cc:3037] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:01.054531 5290 raft_consensus.cc:504] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.054931 5290 leader_election.cc:288] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [CANDIDATE]: Term 1 election: Requested vote from peers 3af918236ad840619f3d13b000e40e5a (127.18.44.193:41433), 7f88d2c1c2b147d68f304af2b95ad053 (127.18.44.195:35217)
I20260923 18:10:01.055665 4958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8db61476dd4d7496340f564af2e6ab" candidate_uuid: "a882e6c3117d4f83b88df623ab870c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af918236ad840619f3d13b000e40e5a"
I20260923 18:10:01.055732 5220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8db61476dd4d7496340f564af2e6ab" candidate_uuid: "a882e6c3117d4f83b88df623ab870c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f88d2c1c2b147d68f304af2b95ad053"
I20260923 18:10:01.055814 4958 raft_consensus.cc:3037] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:01.055950 5220 raft_consensus.cc:3037] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:01.057097 4958 raft_consensus.cc:2454] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a882e6c3117d4f83b88df623ab870c5a in term 1.
I20260923 18:10:01.057411 5220 raft_consensus.cc:2454] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a882e6c3117d4f83b88df623ab870c5a in term 1.
I20260923 18:10:01.057631 5022 leader_election.cc:302] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af918236ad840619f3d13b000e40e5a, a882e6c3117d4f83b88df623ab870c5a; no voters:
I20260923 18:10:01.057901 5290 raft_consensus.cc:2781] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:01.058230 5290 raft_consensus.cc:686] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 1 LEADER]: Becoming Leader. State: Replica: a882e6c3117d4f83b88df623ab870c5a, State: Running, Role: LEADER
I20260923 18:10:01.058728 5290 consensus_queue.cc:237] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } }
I20260923 18:10:01.063195 4816 catalog_manager.cc:5495] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to a882e6c3117d4f83b88df623ab870c5a (127.18.44.194). New cstate: current_term: 1 leader_uuid: "a882e6c3117d4f83b88df623ab870c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a882e6c3117d4f83b88df623ab870c5a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:01.096731 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.090617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.18.44.194:39021)
0923 18:10:01.090845 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:01.090849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:01.090868 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:10:01.090955 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:01.090958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:01.091022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:01.091142 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:01.091678 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.092241 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.094426 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:01.094693 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:01.094831 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:01.095495 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:01.096066 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:01.096339 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:01.096541 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 18:10:01.096593 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:10:01.105192 4958 raft_consensus.cc:1261] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 1 FOLLOWER]: Refusing update from remote peer a882e6c3117d4f83b88df623ab870c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:01.105525 5220 raft_consensus.cc:1261] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 1 FOLLOWER]: Refusing update from remote peer a882e6c3117d4f83b88df623ab870c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:01.106316 5290 consensus_queue.cc:1035] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af918236ad840619f3d13b000e40e5a" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:01.106765 5293 consensus_queue.cc:1035] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f88d2c1c2b147d68f304af2b95ad053" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:01.117686 5297 mvcc.cc:205] Tried to move back new op lower bound from 7332605956514504704 to 7332605956340432896. Current Snapshot: MvccSnapshot[applied={T|T < 7332605956514504704}]
I20260923 18:10:01.130127 4816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:01.130330 4816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:01.133476 4816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab31dcf8863641bbb3f6737e4ce1271f: Sending DeleteTablet for 3 replicas of tablet 6b8db61476dd4d7496340f564af2e6ab
I20260923 18:10:01.134569 4938 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8db61476dd4d7496340f564af2e6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:01 UTC) from {username='slave'} at 127.0.0.1:54442
I20260923 18:10:01.135031 5200 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8db61476dd4d7496340f564af2e6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:01 UTC) from {username='slave'} at 127.0.0.1:33048
I20260923 18:10:01.135155 5069 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8db61476dd4d7496340f564af2e6ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:01 UTC) from {username='slave'} at 127.0.0.1:38110
I20260923 18:10:01.135319 5306 tablet_replica.cc:330] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a: stopping tablet replica
I20260923 18:10:01.135738 5308 tablet_replica.cc:330] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: stopping tablet replica
I20260923 18:10:01.135872 5306 raft_consensus.cc:2229] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:01.136245 5308 raft_consensus.cc:2229] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:01.136366 5306 raft_consensus.cc:2258] T 6b8db61476dd4d7496340f564af2e6ab P 3af918236ad840619f3d13b000e40e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:01.136425 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4876
I20260923 18:10:01.137413 5308 raft_consensus.cc:2258] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:01.139384 5308 ts_tablet_manager.cc:1905] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:01.142432 5308 ts_tablet_manager.cc:1918] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:01.142555 5308 log.cc:1196] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/6b8db61476dd4d7496340f564af2e6ab
I20260923 18:10:01.142832 5308 ts_tablet_manager.cc:1939] T 6b8db61476dd4d7496340f564af2e6ab P 7f88d2c1c2b147d68f304af2b95ad053: Deleting consensus metadata
I20260923 18:10:01.143201 5307 tablet_replica.cc:330] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: stopping tablet replica
I20260923 18:10:01.143831 5307 raft_consensus.cc:2229] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:01.144445 4800 catalog_manager.cc:4841] TS 7f88d2c1c2b147d68f304af2b95ad053 (127.18.44.195:35217): tablet 6b8db61476dd4d7496340f564af2e6ab (table test-table [id=129d7cf28f804542bfe578eb1e5eb6ce]) successfully deleted
I20260923 18:10:01.144518 5307 raft_consensus.cc:2258] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:10:01.145862 4800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:41433 (error 108)
I20260923 18:10:01.146783 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5007
I20260923 18:10:01.146821 5307 ts_tablet_manager.cc:1905] T 6b8db61476dd4d7496340f564af2e6ab P a882e6c3117d4f83b88df623ab870c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:01.146981 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.146613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:41433 (local address 127.0.0.1:54444)
0923 18:10:01.146797 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:01.146892 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260923 18:10:01.148123 4800 catalog_manager.cc:4584] TS 3af918236ad840619f3d13b000e40e5a (127.18.44.193:41433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b8db61476dd4d7496340f564af2e6ab: Network error: Client connection negotiation failed: client connection to 127.18.44.193:41433: connect: Connection refused (error 111)
I20260923 18:10:01.154915 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5138
I20260923 18:10:01.156939 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:01.156469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:39021 (local address 127.0.0.1:38124)
0923 18:10:01.156694 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:01.156795 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:39021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 18:10:01.161862 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 4785
2026-09-23T18:10:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1413 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:01 dist-test-slave-rvjj krb5kdc[5312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:01 dist-test-slave-rvjj krb5kdc[5312](info): set up 2 sockets
Sep 23 18:10:01 dist-test-slave-rvjj krb5kdc[5312](info): commencing operation
krb5kdc: starting...
W20260923 18:10:02.188577 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:02 dist-test-slave-rvjj krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:02.336779 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44289
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:02.487203 5325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:02.487665 5325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:02.487732 5325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:02.492448 5325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:02.492566 5325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:02.492595 5325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:02.492619 5325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:02.492641 5325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:02.499562 5325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:02.501202 5325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:02.502667 5325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:02.513932 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:02.514027 5325 server_base.cc:1034] running on GCE node
W20260923 18:10:02.514021 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:02.514115 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:02.515074 5325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:02.516503 5325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:02.517765 5325 hybrid_clock.cc:648] HybridClock initialized: now 1790187002517738 us; error 56 us; skew 500 ppm
Sep 23 18:10:02 dist-test-slave-rvjj krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:02.521572 5325 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:02.522859 5325 webserver.cc:460] Webserver started at http://127.18.44.254:34941/ using document root <none> and password file <none>
I20260923 18:10:02.523654 5325 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:02.523749 5325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:02.524016 5325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:02.526382 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "41f3e4166fd444e59ca55084ab4e23e4"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "b865ad37e9e415b0c52f8d8345a74fef"
server_key_iv: "c84ca63358674b50929b561ba6eb5a9f"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.527011 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "41f3e4166fd444e59ca55084ab4e23e4"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "b865ad37e9e415b0c52f8d8345a74fef"
server_key_iv: "c84ca63358674b50929b561ba6eb5a9f"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.531674 5325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:10:02.535436 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:02.537242 5325 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:02.537436 5325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "41f3e4166fd444e59ca55084ab4e23e4"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "b865ad37e9e415b0c52f8d8345a74fef"
server_key_iv: "c84ca63358674b50929b561ba6eb5a9f"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.537578 5325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:02.548815 5325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:02.553534 5325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:02.553980 5325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:02.562227 5325 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:44289
I20260923 18:10:02.562286 5392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:44289 every 8 connection(s)
I20260923 18:10:02.563587 5325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:02.566663 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5325
I20260923 18:10:02.566851 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:02.567229 18611 external_mini_cluster.cc:1428] Setting key 924f871dc3ce3f9aef05a7a96f8d65c5
I20260923 18:10:02.568221 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:02.577227 5393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: Bootstrap starting.
Sep 23 18:10:02 dist-test-slave-rvjj krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:02.580672 5393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:02.581619 5393 log.cc:824] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:02.584028 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:02.569308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.18.44.254:44289)
0923 18:10:02.570039 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:02.570061 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:02.570111 (+ 50us) server_negotiation.cc:407] Connection header received
0923 18:10:02.570884 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:02.570906 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:02.571366 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:02.571758 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:02.572707 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.573883 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.574816 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.575418 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.575586 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:02.578519 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:02.578526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:02.578543 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:02.578583 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:02.580396 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:02.580834 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:02.580884 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:02.580889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:02.581020 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:02.581283 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:02.581286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:02.581287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:02.581748 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:02.581942 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:02.582494 (+ 552us) server_negotiation.cc:299] Negotiation successful
0923 18:10:02.582835 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":232,"threads_started":1}
I20260923 18:10:02.584739 5393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: No bootstrap required, opened a new log
I20260923 18:10:02.589733 5393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } }
I20260923 18:10:02.590116 5393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:02.590157 5393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f3e4166fd444e59ca55084ab4e23e4, State: Initialized, Role: FOLLOWER
I20260923 18:10:02.590677 5393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } }
I20260923 18:10:02.590829 5393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:02.590878 5393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:02.590971 5393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:02.592388 5393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } }
I20260923 18:10:02.592859 5393 leader_election.cc:302] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f3e4166fd444e59ca55084ab4e23e4; no voters:
I20260923 18:10:02.593252 5393 leader_election.cc:288] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:02.593677 5398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:02.594003 5398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [term 1 LEADER]: Becoming Leader. State: Replica: 41f3e4166fd444e59ca55084ab4e23e4, State: Running, Role: LEADER
I20260923 18:10:02.594548 5398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } }
I20260923 18:10:02.594957 5393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:02.596621 5399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f3e4166fd444e59ca55084ab4e23e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } } }
I20260923 18:10:02.596863 5399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:02.596665 5400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f3e4166fd444e59ca55084ab4e23e4. Latest consensus state: current_term: 1 leader_uuid: "41f3e4166fd444e59ca55084ab4e23e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f3e4166fd444e59ca55084ab4e23e4" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44289 } } }
I20260923 18:10:02.597084 5400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:02.597414 5410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:02.601732 5410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:02.609983 5410 catalog_manager.cc:1347] Generated new cluster ID: 40f1384e916649d68d65ecb9b6200fef
I20260923 18:10:02.610097 5410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:02.634758 5410 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:02.636169 5410 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:02.647142 5410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: Generated new TSK 0
I20260923 18:10:02.648133 5410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.44.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
Entry for principal kudu/127.18.44.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab.
I20260923 18:10:02.687649 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:02.831990 5419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:02.832296 5419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:02.832342 5419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:02.836557 5419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:02.836661 5419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:02.836724 5419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:10:02.842415 5419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.193.keytab
--principal=kudu/127.18.44.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:02.843652 5419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:02.844784 5419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:02.855237 5424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:02.855393 5425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:02.855382 5427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:02.855722 5419 server_base.cc:1034] running on GCE node
I20260923 18:10:02.856168 5419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:02.856822 5419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:02.858013 5419 hybrid_clock.cc:648] HybridClock initialized: now 1790187002857992 us; error 35 us; skew 500 ppm
Sep 23 18:10:02 dist-test-slave-rvjj krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:02.860955 5419 init.cc:377] Logged in from keytab as kudu/127.18.44.193@KRBTEST.COM (short username kudu)
I20260923 18:10:02.861925 5419 webserver.cc:460] Webserver started at http://127.18.44.193:46443/ using document root <none> and password file <none>
I20260923 18:10:02.862478 5419 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:02.862547 5419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:02.862763 5419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:02.864972 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "64ec8e07e52b418ba126b3d7dc645ebb"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "4235b577c53bc79e74aadb3bc0373dc5"
server_key_iv: "cc1bda3fc624ad7cd86dc3e0394529bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.865568 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "64ec8e07e52b418ba126b3d7dc645ebb"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "4235b577c53bc79e74aadb3bc0373dc5"
server_key_iv: "cc1bda3fc624ad7cd86dc3e0394529bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.869650 5419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:02.872324 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:02.873709 5419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:02.873850 5419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "64ec8e07e52b418ba126b3d7dc645ebb"
format_stamp: "Formatted at 2026-09-23 18:10:02 on dist-test-slave-rvjj"
server_key: "4235b577c53bc79e74aadb3bc0373dc5"
server_key_iv: "cc1bda3fc624ad7cd86dc3e0394529bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:02.873966 5419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:02.884411 5419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:02.887858 5419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:02.888118 5419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:02.888742 5419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:02.889616 5419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:02.889667 5419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:02.889722 5419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:02.889760 5419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:02.898753 5419 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:35353
I20260923 18:10:02.898829 5547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:35353 every 8 connection(s)
I20260923 18:10:02.899770 5419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:02.906189 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5419
I20260923 18:10:02.906335 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:02.906611 18611 external_mini_cluster.cc:1428] Setting key 681f9f5def11edb45e80f111ea1d17ef
Sep 23 18:10:02 dist-test-slave-rvjj krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187002, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.193@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:02.914873 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:02.901905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:43623 (local address 127.18.44.254:44289)
0923 18:10:02.902226 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:02.902231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:02.903673 (+ 1442us) server_negotiation.cc:407] Connection header received
0923 18:10:02.904572 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:02.904576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:02.904635 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:02.904715 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:02.906004 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.906633 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.908082 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.908330 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.908417 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:02.911073 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:02.911076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:02.911078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:02.911106 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:02.912599 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:02.913212 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:02.913216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:02.913219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:02.913362 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:02.913711 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:02.913713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:02.913714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:02.913911 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:02.914000 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:02.914548 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 18:10:02.914701 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
WARNING: no policy specified for kudu/127.18.44.194@KRBTEST.COM; defaulting to no policy
I20260923 18:10:02.915884 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:02.902294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44289 (local address 127.18.44.193:43623)
0923 18:10:02.903462 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:02.903515 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:02.904354 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:02.904927 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:02.904934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:02.905322 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:02.905838 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:02.905853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.906794 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.906797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:02.907970 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:02.907982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:02.908494 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:02.908498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:02.908670 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:02.909308 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:02.909326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:02.910871 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:02.912782 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:02.912790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:02.912809 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:02.913103 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:02.913460 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:02.913463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:02.913464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:02.913618 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:02.914019 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:02.914027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:02.914304 (+ 277us) client_negotiation.cc:769] Sending connection context
0923 18:10:02.914489 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 18:10:02.914873 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":681,"threads_started":1}
I20260923 18:10:02.917510 5548 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44289
I20260923 18:10:02.917768 5548 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:02.918350 5548 heartbeater.cc:511] Master 127.18.44.254:44289 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.194@KRBTEST.COM" created.
I20260923 18:10:02.920231 5355 ts_manager.cc:194] Registered new tserver with Master: 64ec8e07e52b418ba126b3d7dc645ebb (127.18.44.193:35353)
I20260923 18:10:02.921713 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.193@KRBTEST.COM'} at 127.18.44.193:43623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
Entry for principal kudu/127.18.44.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab.
I20260923 18:10:02.940405 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:03.065706 5553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:03.066020 5553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:03.066082 5553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:03.070340 5553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:03.070430 5553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:03.070514 5553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:10:03.076429 5553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.194.keytab
--principal=kudu/127.18.44.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:03.077661 5553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:03.078817 5553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:03.089843 5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:03.089773 5558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:03.089869 5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:03.090240 5553 server_base.cc:1034] running on GCE node
I20260923 18:10:03.090791 5553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:03.091571 5553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:03.092782 5553 hybrid_clock.cc:648] HybridClock initialized: now 1790187003092749 us; error 48 us; skew 500 ppm
Sep 23 18:10:03 dist-test-slave-rvjj krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:03.096067 5553 init.cc:377] Logged in from keytab as kudu/127.18.44.194@KRBTEST.COM (short username kudu)
I20260923 18:10:03.097240 5553 webserver.cc:460] Webserver started at http://127.18.44.194:42307/ using document root <none> and password file <none>
I20260923 18:10:03.097889 5553 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:03.097962 5553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:03.098162 5553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:03.100464 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "17ddf70237ee46a596bcfa17351a8dc9"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "672ee6feee6f1c05ae2614218c3d6e7d"
server_key_iv: "8b44dc323e3c30026be3f98f97329ff0"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.101112 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "17ddf70237ee46a596bcfa17351a8dc9"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "672ee6feee6f1c05ae2614218c3d6e7d"
server_key_iv: "8b44dc323e3c30026be3f98f97329ff0"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.105347 5553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:03.108364 5568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.109876 5553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:03.110056 5553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "17ddf70237ee46a596bcfa17351a8dc9"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "672ee6feee6f1c05ae2614218c3d6e7d"
server_key_iv: "8b44dc323e3c30026be3f98f97329ff0"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.110183 5553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:03.139497 5553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:03.142804 5553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:03.143010 5553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:03.143658 5553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:03.144546 5553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:03.144595 5553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.144651 5553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:03.144677 5553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.156674 5553 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:37515
I20260923 18:10:03.156692 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:37515 every 8 connection(s)
I20260923 18:10:03.157686 5553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:03.158640 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5553
I20260923 18:10:03.158911 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:03.159539 18611 external_mini_cluster.cc:1428] Setting key 4d04ccd4c445362f840c3e0ba6174457
Sep 23 18:10:03 dist-test-slave-rvjj krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.194@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.44.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.195@KRBTEST.COM" created.
I20260923 18:10:03.175789 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.159989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:45829 (local address 127.18.44.254:44289)
0923 18:10:03.160114 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.160119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.161161 (+ 1042us) server_negotiation.cc:407] Connection header received
0923 18:10:03.162980 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.162983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.163036 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.163123 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:03.164796 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.165472 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.167582 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.167966 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.168106 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.170854 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:03.170857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:03.170860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:03.170889 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:03.172747 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.173552 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.173557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.173558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.173662 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.174190 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.174194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.174196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.174508 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:03.174600 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.175451 (+ 851us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.175612 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 18:10:03.176875 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.160443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44289 (local address 127.18.44.194:45829)
0923 18:10:03.160984 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.161053 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.162628 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.163416 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.163427 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.163989 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:03.164586 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.164604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.165643 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.165648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.167391 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.167409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.168190 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.168194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.168378 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.169041 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:03.169060 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:03.170616 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:03.172932 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:03.172939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:03.172957 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:03.173333 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:03.173792 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:03.173796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:03.173797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:03.174039 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:03.174633 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:03.174642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:03.175130 (+ 488us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.175373 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.175771 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260923 18:10:03.178665 5682 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44289
I20260923 18:10:03.179023 5682 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:03.179711 5682 heartbeater.cc:511] Master 127.18.44.254:44289 requested a full tablet report, sending...
I20260923 18:10:03.181387 5355 ts_manager.cc:194] Registered new tserver with Master: 17ddf70237ee46a596bcfa17351a8dc9 (127.18.44.194:37515)
I20260923 18:10:03.182547 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.194@KRBTEST.COM'} at 127.18.44.194:45829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
Entry for principal kudu/127.18.44.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab.
I20260923 18:10:03.201479 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:03.325603 5687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:03.325956 5687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:03.326004 5687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:03.330986 5687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:03.331094 5687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:03.331159 5687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:10:03.337108 5687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.195.keytab
--principal=kudu/127.18.44.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:03.338312 5687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:03.339443 5687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:03.351286 5695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:03.351243 5692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:03.351377 5693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:03.351585 5687 server_base.cc:1034] running on GCE node
I20260923 18:10:03.352028 5687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:03.352730 5687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:03.353926 5687 hybrid_clock.cc:648] HybridClock initialized: now 1790187003353896 us; error 47 us; skew 500 ppm
Sep 23 18:10:03 dist-test-slave-rvjj krb5kdc[5312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:03.357226 5687 init.cc:377] Logged in from keytab as kudu/127.18.44.195@KRBTEST.COM (short username kudu)
I20260923 18:10:03.358363 5687 webserver.cc:460] Webserver started at http://127.18.44.195:42011/ using document root <none> and password file <none>
I20260923 18:10:03.358974 5687 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:03.359053 5687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:03.359319 5687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:03.361600 5687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "71123570e22243f1ac36364e455b2cce"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "1f0a8c11608e08edcc52452625c8105a"
server_key_iv: "94505e3410b98420887914192468ff40"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.362515 5687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "71123570e22243f1ac36364e455b2cce"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "1f0a8c11608e08edcc52452625c8105a"
server_key_iv: "94505e3410b98420887914192468ff40"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.367038 5687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 18:10:03.370150 5702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.371855 5687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:03.372001 5687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "71123570e22243f1ac36364e455b2cce"
format_stamp: "Formatted at 2026-09-23 18:10:03 on dist-test-slave-rvjj"
server_key: "1f0a8c11608e08edcc52452625c8105a"
server_key_iv: "94505e3410b98420887914192468ff40"
server_key_version: "encryptionkey@0"
I20260923 18:10:03.372128 5687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:03.408807 5687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:03.413805 5687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:03.414095 5687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:03.414892 5687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:03.416224 5687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:03.416299 5687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.416363 5687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:03.416384 5687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:03.427894 5687 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:40545
I20260923 18:10:03.427911 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:40545 every 8 connection(s)
I20260923 18:10:03.429183 5687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:03.429716 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5687
I20260923 18:10:03.429873 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:03.430207 18611 external_mini_cluster.cc:1428] Setting key 3520a63b4aa422c7e6786f0c0fe23a70
Sep 23 18:10:03 dist-test-slave-rvjj krb5kdc[5312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.195@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:03.446872 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.432140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:51749 (local address 127.18.44.254:44289)
0923 18:10:03.432372 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.432376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.433436 (+ 1060us) server_negotiation.cc:407] Connection header received
0923 18:10:03.435002 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.435005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.435060 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.435154 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:03.436799 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.437347 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.439215 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.439528 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.439688 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.442961 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:03.442964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:03.442967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:03.442998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:03.444644 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.445235 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.445237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.445238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.445323 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.445705 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.445709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.445711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.445924 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:03.446015 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.446624 (+ 609us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.446748 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260923 18:10:03.447824 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.432682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:44289 (local address 127.18.44.195:51749)
0923 18:10:03.433276 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.433338 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.434697 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.435574 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.435583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.436015 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:03.436574 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.436591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.437592 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.437595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.438926 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.438998 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.439743 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.439747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.439947 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.440785 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:03.440807 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:03.442666 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:03.444781 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:03.444789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:03.444819 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:03.445135 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:03.445418 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:03.445421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:03.445424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:03.445609 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:03.446042 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:03.446049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:03.446349 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.446539 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.446888 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":179,"threads_started":1}
I20260923 18:10:03.449421 5816 heartbeater.cc:347] Connected to a master server at 127.18.44.254:44289
I20260923 18:10:03.449712 5816 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:03.450356 5816 heartbeater.cc:511] Master 127.18.44.254:44289 requested a full tablet report, sending...
I20260923 18:10:03.451793 5355 ts_manager.cc:194] Registered new tserver with Master: 71123570e22243f1ac36364e455b2cce (127.18.44.195:40545)
I20260923 18:10:03.452409 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.44.195@KRBTEST.COM'} at 127.18.44.195:51749
I20260923 18:10:03.458050 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:03.468065 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.460481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.18.44.254:44289)
0923 18:10:03.460667 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.460673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.460854 (+ 181us) server_negotiation.cc:407] Connection header received
0923 18:10:03.460975 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.460978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.461022 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.461105 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:03.461825 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.462361 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.463237 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.463503 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.463593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.464906 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:03.464910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:03.464912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:03.464939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:03.466402 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.466814 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.466817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.466819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.466898 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:03.467155 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:03.467158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:03.467160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:03.467455 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:03.467580 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.467791 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.467905 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 18:10:03.471241 5355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:03.474514 5355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:03.490413 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.483921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.18.44.193:35353)
0923 18:10:03.484385 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.484393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.484420 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:10:03.484672 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.484678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.484935 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.485089 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:03.485602 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.486959 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.488611 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.489558 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.489667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.489842 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.490126 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.490250 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260923 18:10:03.490398 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.483868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35353 (local address 127.0.0.1:58632)
0923 18:10:03.484279 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.484331 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.484460 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.485285 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.485291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.485318 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:03.485461 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.485469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.487068 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.487071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.488448 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.488459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.489697 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.489701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.489835 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.489924 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.490075 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.490226 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":26,"thread_start_us":117,"threads_started":1}
I20260923 18:10:03.490983 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.484654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40545 (local address 127.0.0.1:51094)
0923 18:10:03.485610 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.485623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.485702 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.486282 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.486285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.486305 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:03.486416 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.486424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.487940 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.487944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.489099 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.489109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.490387 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.490390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.490478 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.490655 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.490727 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.490808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260923 18:10:03.491434 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.484702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.18.44.194:37515)
0923 18:10:03.485151 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.485162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.485615 (+ 453us) server_negotiation.cc:407] Connection header received
0923 18:10:03.485921 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.485929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.486130 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.486307 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:03.486768 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.487969 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.489547 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.490327 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.490433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.490570 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.491130 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.491236 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":157,"threads_started":1}
I20260923 18:10:03.491814 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.485077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.18.44.195:40545)
0923 18:10:03.485685 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.485699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.485746 (+ 47us) server_negotiation.cc:407] Connection header received
0923 18:10:03.485902 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.485908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.486128 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.486294 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:03.486535 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.487729 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.489239 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.490064 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.490169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.490321 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.491385 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.491676 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":96,"threads_started":1}
I20260923 18:10:03.491030 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.483903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:37515 (local address 127.0.0.1:51480)
0923 18:10:03.485306 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.485320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.485694 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.486531 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.486534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.486552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:03.486639 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.486644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.488085 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.488088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.489387 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.489398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.490640 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.490644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.490730 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.490751 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.490814 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.490889 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":24,"thread_start_us":129,"threads_started":1}
I20260923 18:10:03.494156 5482 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b51c269e34120bff82cba0430806a (DEFAULT_TABLE table=test-table [id=416e28a8049445a39ca0c5dd357caf8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:03.494335 5750 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b51c269e34120bff82cba0430806a (DEFAULT_TABLE table=test-table [id=416e28a8049445a39ca0c5dd357caf8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:03.494702 5597 tablet_service.cc:1467] Processing CreateTablet for tablet ad9b51c269e34120bff82cba0430806a (DEFAULT_TABLE table=test-table [id=416e28a8049445a39ca0c5dd357caf8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:03.495440 5750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9b51c269e34120bff82cba0430806a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:03.495488 5482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9b51c269e34120bff82cba0430806a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:03.495700 5597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9b51c269e34120bff82cba0430806a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:03.503013 5834 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Bootstrap starting.
I20260923 18:10:03.503618 5835 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Bootstrap starting.
I20260923 18:10:03.503618 5833 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Bootstrap starting.
I20260923 18:10:03.505576 5834 tablet_bootstrap.cc:654] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:03.505863 5835 tablet_bootstrap.cc:654] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:03.505856 5833 tablet_bootstrap.cc:654] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:03.506374 5834 log.cc:824] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:03.507138 5835 log.cc:824] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:03.507146 5833 log.cc:824] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:03.508597 5834 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: No bootstrap required, opened a new log
I20260923 18:10:03.508803 5834 ts_tablet_manager.cc:1397] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:10:03.509202 5833 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: No bootstrap required, opened a new log
I20260923 18:10:03.509398 5833 ts_tablet_manager.cc:1397] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:10:03.509503 5835 tablet_bootstrap.cc:492] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: No bootstrap required, opened a new log
I20260923 18:10:03.509685 5835 ts_tablet_manager.cc:1397] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:10:03.512740 5834 raft_consensus.cc:348] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.513028 5834 raft_consensus.cc:374] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:03.513087 5834 raft_consensus.cc:729] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ec8e07e52b418ba126b3d7dc645ebb, State: Initialized, Role: FOLLOWER
I20260923 18:10:03.513434 5835 raft_consensus.cc:348] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.513434 5833 raft_consensus.cc:348] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.513590 5834 consensus_queue.cc:260] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.513695 5835 raft_consensus.cc:374] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:03.513695 5833 raft_consensus.cc:374] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:03.513754 5833 raft_consensus.cc:729] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71123570e22243f1ac36364e455b2cce, State: Initialized, Role: FOLLOWER
I20260923 18:10:03.513754 5835 raft_consensus.cc:729] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ddf70237ee46a596bcfa17351a8dc9, State: Initialized, Role: FOLLOWER
I20260923 18:10:03.514220 5835 consensus_queue.cc:260] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.514349 5548 heartbeater.cc:503] Master 127.18.44.254:44289 was elected leader, sending a full tablet report...
I20260923 18:10:03.514698 5834 ts_tablet_manager.cc:1428] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:10:03.514863 5833 consensus_queue.cc:260] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.515306 5682 heartbeater.cc:503] Master 127.18.44.254:44289 was elected leader, sending a full tablet report...
I20260923 18:10:03.515793 5833 ts_tablet_manager.cc:1428] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260923 18:10:03.516027 5816 heartbeater.cc:503] Master 127.18.44.254:44289 was elected leader, sending a full tablet report...
I20260923 18:10:03.515300 5835 ts_tablet_manager.cc:1428] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260923 18:10:03.651541 5549 tablet.cc:2369] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:03.659924 5683 tablet.cc:2369] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:03.680570 5817 tablet.cc:2369] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:10:03.947124 5840 raft_consensus.cc:482] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:03.947471 5840 raft_consensus.cc:504] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.948760 5840 leader_election.cc:288] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ec8e07e52b418ba126b3d7dc645ebb (127.18.44.193:35353), 71123570e22243f1ac36364e455b2cce (127.18.44.195:40545)
I20260923 18:10:03.952740 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.949056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:40545 (local address 127.18.44.194:37215)
0923 18:10:03.949437 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.949450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.949594 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.949831 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.949833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.949850 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:03.950008 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.950014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.950733 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.950736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.951698 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.951710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.952354 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.952357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.952454 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.952473 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.952521 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.952588 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":61,"spinlock_wait_cycles":16640,"thread_start_us":64,"threads_started":1}
I20260923 18:10:03.953016 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.949210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:37215 (local address 127.18.44.195:40545)
0923 18:10:03.949339 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.949343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.949538 (+ 195us) server_negotiation.cc:407] Connection header received
0923 18:10:03.949707 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.949710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.949753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.949816 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:03.950113 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.950617 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.951830 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.952235 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.952440 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.952591 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.952686 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.952829 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:10:03.953244 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.949100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:57451 (local address 127.18.44.193:35353)
0923 18:10:03.949264 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:03.949268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:03.949541 (+ 273us) server_negotiation.cc:407] Connection header received
0923 18:10:03.949702 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:03.949706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:03.949753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:03.949828 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:03.950124 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.950631 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.951830 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.952246 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.952354 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.952412 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:03.952961 (+ 549us) server_negotiation.cc:299] Negotiation successful
0923 18:10:03.953113 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 18:10:03.953712 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:03.949000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35353 (local address 127.18.44.194:57451)
0923 18:10:03.949442 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:03.949454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:03.949594 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:03.949850 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:03.949853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:03.949869 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:03.950020 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.950026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.950737 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.950740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.951707 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:03.951718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:03.952480 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:03.952486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:03.952669 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:03.952700 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 18:10:03.953412 (+ 712us) client_negotiation.cc:240] Negotiation successful
0923 18:10:03.953535 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":4,"spinlock_wait_cycles":28672,"thread_start_us":110,"threads_started":1}
I20260923 18:10:03.954037 5770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b51c269e34120bff82cba0430806a" candidate_uuid: "17ddf70237ee46a596bcfa17351a8dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71123570e22243f1ac36364e455b2cce" is_pre_election: true
I20260923 18:10:03.954502 5770 raft_consensus.cc:2454] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ddf70237ee46a596bcfa17351a8dc9 in term 0.
I20260923 18:10:03.954509 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b51c269e34120bff82cba0430806a" candidate_uuid: "17ddf70237ee46a596bcfa17351a8dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" is_pre_election: true
I20260923 18:10:03.955009 5502 raft_consensus.cc:2454] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ddf70237ee46a596bcfa17351a8dc9 in term 0.
I20260923 18:10:03.955238 5572 leader_election.cc:302] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ddf70237ee46a596bcfa17351a8dc9, 71123570e22243f1ac36364e455b2cce; no voters:
I20260923 18:10:03.955727 5840 raft_consensus.cc:2781] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:03.955853 5840 raft_consensus.cc:482] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:03.955893 5840 raft_consensus.cc:3037] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:03.957091 5840 raft_consensus.cc:504] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.957585 5840 leader_election.cc:288] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [CANDIDATE]: Term 1 election: Requested vote from peers 64ec8e07e52b418ba126b3d7dc645ebb (127.18.44.193:35353), 71123570e22243f1ac36364e455b2cce (127.18.44.195:40545)
I20260923 18:10:03.958295 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b51c269e34120bff82cba0430806a" candidate_uuid: "17ddf70237ee46a596bcfa17351a8dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ec8e07e52b418ba126b3d7dc645ebb"
I20260923 18:10:03.958295 5770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9b51c269e34120bff82cba0430806a" candidate_uuid: "17ddf70237ee46a596bcfa17351a8dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71123570e22243f1ac36364e455b2cce"
I20260923 18:10:03.958530 5502 raft_consensus.cc:3037] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:03.958530 5770 raft_consensus.cc:3037] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:03.960376 5502 raft_consensus.cc:2454] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ddf70237ee46a596bcfa17351a8dc9 in term 1.
I20260923 18:10:03.960376 5770 raft_consensus.cc:2454] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ddf70237ee46a596bcfa17351a8dc9 in term 1.
I20260923 18:10:03.961149 5572 leader_election.cc:302] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ddf70237ee46a596bcfa17351a8dc9, 71123570e22243f1ac36364e455b2cce; no voters:
I20260923 18:10:03.961478 5840 raft_consensus.cc:2781] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:03.961782 5840 raft_consensus.cc:686] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 17ddf70237ee46a596bcfa17351a8dc9, State: Running, Role: LEADER
I20260923 18:10:03.962155 5840 consensus_queue.cc:237] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } }
I20260923 18:10:03.966549 5354 catalog_manager.cc:5495] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17ddf70237ee46a596bcfa17351a8dc9 (127.18.44.194). New cstate: current_term: 1 leader_uuid: "17ddf70237ee46a596bcfa17351a8dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17ddf70237ee46a596bcfa17351a8dc9" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 37515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:04.067114 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.062443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.18.44.194:37515)
0923 18:10:04.062752 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:04.062757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:04.062775 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:10:04.062888 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:04.062890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:04.062974 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:04.063128 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:04.063451 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.064084 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.065098 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.065358 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.065454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:04.065897 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:04.066384 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:04.066625 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:04.066792 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 18:10:04.066873 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260923 18:10:04.075511 5770 raft_consensus.cc:1261] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 1 FOLLOWER]: Refusing update from remote peer 17ddf70237ee46a596bcfa17351a8dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:04.075664 5502 raft_consensus.cc:1261] T ad9b51c269e34120bff82cba0430806a P 64ec8e07e52b418ba126b3d7dc645ebb [term 1 FOLLOWER]: Refusing update from remote peer 17ddf70237ee46a596bcfa17351a8dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:04.077198 5840 consensus_queue.cc:1035] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ec8e07e52b418ba126b3d7dc645ebb" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:04.077703 5844 consensus_queue.cc:1035] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71123570e22243f1ac36364e455b2cce" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:04.089627 5853 mvcc.cc:205] Tried to move back new op lower bound from 7332605968682835968 to 7332605968232243200. Current Snapshot: MvccSnapshot[applied={T|T < 7332605968682835968}]
I20260923 18:10:04.090857 5852 mvcc.cc:205] Tried to move back new op lower bound from 7332605968682835968 to 7332605968232243200. Current Snapshot: MvccSnapshot[applied={T|T < 7332605968682835968}]
I20260923 18:10:04.099084 5354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:04.099407 5354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:04.102865 5354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41f3e4166fd444e59ca55084ab4e23e4: Sending DeleteTablet for 3 replicas of tablet ad9b51c269e34120bff82cba0430806a
I20260923 18:10:04.104056 5482 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9b51c269e34120bff82cba0430806a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:04 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:58632
I20260923 18:10:04.104353 5597 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9b51c269e34120bff82cba0430806a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:04 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:51480
I20260923 18:10:04.104712 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5419
I20260923 18:10:04.104810 5750 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9b51c269e34120bff82cba0430806a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:04 UTC) from {username='kudu', principal='kudu/127.18.44.254@KRBTEST.COM'} at 127.0.0.1:51094
I20260923 18:10:04.105623 5861 tablet_replica.cc:330] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: stopping tablet replica
I20260923 18:10:04.106355 5861 raft_consensus.cc:2229] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:04.106863 5861 raft_consensus.cc:2258] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:04.107710 5862 tablet_replica.cc:330] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: stopping tablet replica
I20260923 18:10:04.108251 5862 raft_consensus.cc:2229] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:04.108915 5862 raft_consensus.cc:2258] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:04.111125 5862 ts_tablet_manager.cc:1905] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:04.114048 5861 ts_tablet_manager.cc:1905] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:04.115828 5862 ts_tablet_manager.cc:1918] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260923 18:10:04.115805 5343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:35353 (error 108)
I20260923 18:10:04.116227 5862 log.cc:1196] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/ad9b51c269e34120bff82cba0430806a
I20260923 18:10:04.116776 5862 ts_tablet_manager.cc:1939] T ad9b51c269e34120bff82cba0430806a P 71123570e22243f1ac36364e455b2cce: Deleting consensus metadata
I20260923 18:10:04.117365 5861 ts_tablet_manager.cc:1918] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:04.117518 5861 log.cc:1196] T ad9b51c269e34120bff82cba0430806a P 17ddf70237ee46a596bcfa17351a8dc9: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/ad9b51c269e34120bff82cba0430806a
I20260923 18:10:04.117753 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5553
I20260923 18:10:04.119827 5344 catalog_manager.cc:4841] TS 71123570e22243f1ac36364e455b2cce (127.18.44.195:40545): tablet ad9b51c269e34120bff82cba0430806a (table test-table [id=416e28a8049445a39ca0c5dd357caf8e]) successfully deleted
I20260923 18:10:04.120627 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.117302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:35353 (local address 127.0.0.1:58648)
0923 18:10:04.120410 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:04.120511 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3017,"thread_start_us":97,"threads_started":1}
W20260923 18:10:04.120900 5343 catalog_manager.cc:4584] TS 64ec8e07e52b418ba126b3d7dc645ebb (127.18.44.193:35353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad9b51c269e34120bff82cba0430806a: Network error: Client connection negotiation failed: client connection to 127.18.44.193:35353: connect: Connection refused (error 111)
I20260923 18:10:04.125128 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5687
I20260923 18:10:04.126909 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.125409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:37515 (local address 127.0.0.1:51506)
0923 18:10:04.126713 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:04.126799 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:37515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1214}
I20260923 18:10:04.131330 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5325
2026-09-23T18:10:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T18:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:04Z Disabled control of system clock
I20260923 18:10:04.172704 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:34503
--webserver_interface=127.18.44.254
--webserver_port=0
--builtin_ntp_servers=127.18.44.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:34503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:10:04.335358 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:04.335816 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:04.335860 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:04.341758 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:04.341909 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:04.341928 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:04.341945 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:04.341962 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:04.350605 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:34503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:34503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:04.352810 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:04.355398 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:04.369063 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.369064 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.369529 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:04.369864 5868 server_base.cc:1034] running on GCE node
I20260923 18:10:04.370805 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:04.372570 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:04.373835 5868 hybrid_clock.cc:648] HybridClock initialized: now 1790187004373797 us; error 75 us; skew 500 ppm
I20260923 18:10:04.377174 5868 webserver.cc:460] Webserver started at http://127.18.44.254:37307/ using document root <none> and password file <none>
I20260923 18:10:04.378082 5868 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:04.378185 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:04.378437 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:04.381871 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "bd49696c76274f2e9d9bf19bb5186897"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "386c10a94171757db7849c9120dfc8c8"
server_key_iv: "dba48b4cafd430c95951ced5d3ef0e7c"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.382798 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "bd49696c76274f2e9d9bf19bb5186897"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "386c10a94171757db7849c9120dfc8c8"
server_key_iv: "dba48b4cafd430c95951ced5d3ef0e7c"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.389354 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 18:10:04.394106 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.396097 5868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:10:04.396512 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "bd49696c76274f2e9d9bf19bb5186897"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "386c10a94171757db7849c9120dfc8c8"
server_key_iv: "dba48b4cafd430c95951ced5d3ef0e7c"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.396740 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:04.451730 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:04.452811 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:04.453037 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:04.464586 5868 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:34503
I20260923 18:10:04.464665 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:34503 every 8 connection(s)
I20260923 18:10:04.465992 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:04.469897 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:04.475507 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5868
I20260923 18:10:04.475668 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:04.475944 18611 external_mini_cluster.cc:1428] Setting key 12463a836b5b5f579daeb6bb0af5e2e2
I20260923 18:10:04.476357 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: Bootstrap starting.
I20260923 18:10:04.479195 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:04.480046 5935 log.cc:824] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:04.482307 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: No bootstrap required, opened a new log
I20260923 18:10:04.488147 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } }
I20260923 18:10:04.488476 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:04.488512 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd49696c76274f2e9d9bf19bb5186897, State: Initialized, Role: FOLLOWER
I20260923 18:10:04.489391 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } }
I20260923 18:10:04.489616 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:04.489691 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:04.489823 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:04.491001 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } }
I20260923 18:10:04.491567 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.478029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.18.44.254:34503)
0923 18:10:04.478924 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:04.478942 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:04.478995 (+ 53us) server_negotiation.cc:407] Connection header received
0923 18:10:04.479932 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:04.479959 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:04.480570 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:04.480985 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:04.481464 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.482845 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.484864 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.485754 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.486128 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:04.488039 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:04.488047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:04.488073 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:04.489028 (+ 955us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:04.489161 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:04.489199 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:04.489342 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:04.489735 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 18:10:04.490213 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":168,"threads_started":1}
I20260923 18:10:04.491565 5935 leader_election.cc:302] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd49696c76274f2e9d9bf19bb5186897; no voters:
I20260923 18:10:04.492344 5935 leader_election.cc:288] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:04.492687 5940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:04.493279 5940 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [term 1 LEADER]: Becoming Leader. State: Replica: bd49696c76274f2e9d9bf19bb5186897, State: Running, Role: LEADER
I20260923 18:10:04.493826 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:04.493808 5940 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } }
I20260923 18:10:04.496740 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd49696c76274f2e9d9bf19bb5186897. Latest consensus state: current_term: 1 leader_uuid: "bd49696c76274f2e9d9bf19bb5186897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } } }
I20260923 18:10:04.497354 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:04.497040 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd49696c76274f2e9d9bf19bb5186897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd49696c76274f2e9d9bf19bb5186897" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 34503 } } }
I20260923 18:10:04.497658 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:04.500162 5954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:04.500303 5954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:04.500396 5955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:04.501155 5955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:04.510203 5955 catalog_manager.cc:1347] Generated new cluster ID: 646a13a4117b4038ba2370ce30f25a88
I20260923 18:10:04.510514 5955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:04.527781 5955 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:04.529608 5955 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:04.542264 5955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: Generated new TSK 0
I20260923 18:10:04.543469 5955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:04.554278 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.193:0
--local_ip_for_outbound_sockets=127.18.44.193
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--builtin_ntp_servers=127.18.44.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:10:04.707978 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:04.708314 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:04.708357 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:04.712734 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:04.712821 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:04.712880 5959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.193
I20260923 18:10:04.718335 5959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.44.193
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.193
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:04.719691 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:04.721611 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:04.733340 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.733419 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.733340 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:04.733585 5959 server_base.cc:1034] running on GCE node
I20260923 18:10:04.734194 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:04.734968 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:04.736193 5959 hybrid_clock.cc:648] HybridClock initialized: now 1790187004736165 us; error 45 us; skew 500 ppm
I20260923 18:10:04.738662 5959 webserver.cc:460] Webserver started at http://127.18.44.193:37215/ using document root <none> and password file <none>
I20260923 18:10:04.739400 5959 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:04.739486 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:04.739706 5959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:04.742098 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data/instance:
uuid: "d0e3ebb5a59142d49688dec399d23cf6"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "5212b972f4a0e8e06cf5a61c719ba0e3"
server_key_iv: "6e4ccdde85ff390438ce5cc7c85ee670"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.742761 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance:
uuid: "d0e3ebb5a59142d49688dec399d23cf6"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "5212b972f4a0e8e06cf5a61c719ba0e3"
server_key_iv: "6e4ccdde85ff390438ce5cc7c85ee670"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.747247 5959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:10:04.750315 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.752063 5959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:04.752208 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal
uuid: "d0e3ebb5a59142d49688dec399d23cf6"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "5212b972f4a0e8e06cf5a61c719ba0e3"
server_key_iv: "6e4ccdde85ff390438ce5cc7c85ee670"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.752338 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:04.769035 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:04.769773 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:04.769932 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:04.770543 5959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:04.771610 5959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:04.771662 5959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.771706 5959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:04.771721 5959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.781363 5959 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.193:44567
I20260923 18:10:04.781453 6086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.193:44567 every 8 connection(s)
I20260923 18:10:04.782420 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:04.784789 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 5959
I20260923 18:10:04.784946 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:04.785243 18611 external_mini_cluster.cc:1428] Setting key 78389358de8ac2ca46df8c365bb18ac9
I20260923 18:10:04.787712 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.194:0
--local_ip_for_outbound_sockets=127.18.44.194
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--builtin_ntp_servers=127.18.44.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:10:04.794529 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.784618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.193:58469 (local address 127.18.44.254:34503)
0923 18:10:04.784754 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:04.784758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:04.786402 (+ 1644us) server_negotiation.cc:407] Connection header received
0923 18:10:04.788200 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:04.788203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:04.788263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:04.788354 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:04.789467 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.790015 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.791755 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.792092 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.792192 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:04.793790 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:04.793794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:04.793796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:04.793813 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:04.793838 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:04.793849 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:04.793928 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:04.794172 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 18:10:04.794257 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:10:04.795998 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:04.785193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34503 (local address 127.18.44.193:58469)
0923 18:10:04.786196 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:04.786264 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:04.787957 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:04.788526 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:04.788534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:04.788633 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:04.789296 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:04.789312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.790168 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.790172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:04.791571 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:04.791589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:04.792303 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:04.792309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:04.792569 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:04.793332 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:04.793335 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:04.793420 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:04.793533 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:04.793954 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:04.793965 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:04.793971 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:10:04.794154 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 18:10:04.794601 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":178,"threads_started":1}
I20260923 18:10:04.797891 6087 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34503
I20260923 18:10:04.798195 6087 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:04.798831 6087 heartbeater.cc:511] Master 127.18.44.254:34503 requested a full tablet report, sending...
I20260923 18:10:04.800990 5899 ts_manager.cc:194] Registered new tserver with Master: d0e3ebb5a59142d49688dec399d23cf6 (127.18.44.193:44567)
I20260923 18:10:04.803056 5899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.193:58469
W20260923 18:10:04.922118 6090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:04.922461 6090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:04.922503 6090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:04.927680 6090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:04.927773 6090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:04.927835 6090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.194
I20260923 18:10:04.933215 6090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.44.194
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.194
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:04.934333 6090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:04.935734 6090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:04.946696 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.946717 6098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:04.946719 6095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:04.947117 6090 server_base.cc:1034] running on GCE node
I20260923 18:10:04.947585 6090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:04.948276 6090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:04.949509 6090 hybrid_clock.cc:648] HybridClock initialized: now 1790187004949478 us; error 42 us; skew 500 ppm
I20260923 18:10:04.951854 6090 webserver.cc:460] Webserver started at http://127.18.44.194:36375/ using document root <none> and password file <none>
I20260923 18:10:04.952479 6090 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:04.952560 6090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:04.952795 6090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:04.954957 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data/instance:
uuid: "3a5c064753af4792a940437b2ab7ff8a"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "67cc3e6d05b5b76b2574048e3e577669"
server_key_iv: "75ab3f773c48dad6b6df72faed208346"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.955577 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5c064753af4792a940437b2ab7ff8a"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "67cc3e6d05b5b76b2574048e3e577669"
server_key_iv: "75ab3f773c48dad6b6df72faed208346"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.959729 6090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:04.962595 6104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.964044 6090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:04.964205 6090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal
uuid: "3a5c064753af4792a940437b2ab7ff8a"
format_stamp: "Formatted at 2026-09-23 18:10:04 on dist-test-slave-rvjj"
server_key: "67cc3e6d05b5b76b2574048e3e577669"
server_key_iv: "75ab3f773c48dad6b6df72faed208346"
server_key_version: "encryptionkey@0"
I20260923 18:10:04.964372 6090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:04.989954 6090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:04.991158 6090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:04.991430 6090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:04.992130 6090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:04.994357 6090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:04.994505 6090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:04.994617 6090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:04.994647 6090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:05.007936 6090 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.194:36119
I20260923 18:10:05.007953 6217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.194:36119 every 8 connection(s)
I20260923 18:10:05.008935 6090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:05.016925 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6090
I20260923 18:10:05.017120 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:05.017370 18611 external_mini_cluster.cc:1428] Setting key 4de614472f9f9d410f5e2ea4147d5c43
I20260923 18:10:05.019930 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.44.195:0
--local_ip_for_outbound_sockets=127.18.44.195
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--builtin_ntp_servers=127.18.44.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:10:05.020838 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.011090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.194:54589 (local address 127.18.44.254:34503)
0923 18:10:05.011372 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.011377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.012811 (+ 1434us) server_negotiation.cc:407] Connection header received
0923 18:10:05.014655 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.014659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.014726 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.014807 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:05.015970 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.016516 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.017897 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.018221 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.018324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.019861 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:05.019865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:05.019868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:05.019884 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:05.019910 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:05.019920 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:05.020189 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.020527 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.020606 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260923 18:10:05.021981 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.011502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34503 (local address 127.18.44.194:54589)
0923 18:10:05.012580 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.012636 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.014283 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.015074 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.015084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.015183 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:05.015796 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.015813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.016669 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.016673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.017711 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.017725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.018472 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.018477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.018706 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.019515 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:05.019517 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:05.019573 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:05.019674 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:05.020239 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:05.020256 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:05.020264 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.020495 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.020998 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":154,"threads_started":1}
I20260923 18:10:05.023520 6218 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34503
I20260923 18:10:05.023828 6218 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:05.024552 6218 heartbeater.cc:511] Master 127.18.44.254:34503 requested a full tablet report, sending...
I20260923 18:10:05.026093 5899 ts_manager.cc:194] Registered new tserver with Master: 3a5c064753af4792a940437b2ab7ff8a (127.18.44.194:36119)
I20260923 18:10:05.026700 5899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.194:54589
W20260923 18:10:05.145727 6221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:05.146070 6221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:05.146168 6221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:05.153105 6221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:05.153208 6221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:05.153297 6221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.44.195
I20260923 18:10:05.160153 6221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.44.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.44.195
--webserver_port=0
--tserver_master_addrs=127.18.44.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.44.195
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:05.161346 6221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:05.162807 6221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:05.174194 6227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:05.174386 6221 server_base.cc:1034] running on GCE node
W20260923 18:10:05.174194 6226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:05.174221 6229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:05.175012 6221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:05.175796 6221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:05.177090 6221 hybrid_clock.cc:648] HybridClock initialized: now 1790187005177073 us; error 33 us; skew 500 ppm
I20260923 18:10:05.179705 6221 webserver.cc:460] Webserver started at http://127.18.44.195:43877/ using document root <none> and password file <none>
I20260923 18:10:05.180335 6221 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:05.180416 6221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:05.180637 6221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:05.182934 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data/instance:
uuid: "49a65fb53fe24d68a35c1f3618ed161f"
format_stamp: "Formatted at 2026-09-23 18:10:05 on dist-test-slave-rvjj"
server_key: "b8002d66a1e595231b38282a56f40d75"
server_key_iv: "310d26a56916118662a35efabbd78700"
server_key_version: "encryptionkey@0"
I20260923 18:10:05.183660 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance:
uuid: "49a65fb53fe24d68a35c1f3618ed161f"
format_stamp: "Formatted at 2026-09-23 18:10:05 on dist-test-slave-rvjj"
server_key: "b8002d66a1e595231b38282a56f40d75"
server_key_iv: "310d26a56916118662a35efabbd78700"
server_key_version: "encryptionkey@0"
I20260923 18:10:05.188740 6221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:05.192530 6235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:05.194532 6221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:05.194700 6221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal
uuid: "49a65fb53fe24d68a35c1f3618ed161f"
format_stamp: "Formatted at 2026-09-23 18:10:05 on dist-test-slave-rvjj"
server_key: "b8002d66a1e595231b38282a56f40d75"
server_key_iv: "310d26a56916118662a35efabbd78700"
server_key_version: "encryptionkey@0"
I20260923 18:10:05.194881 6221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:05.209980 6221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:05.210757 6221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:05.210951 6221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:05.211648 6221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:05.212661 6221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:05.212716 6221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:05.212776 6221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:05.212805 6221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:05.221908 6221 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.195:34711
I20260923 18:10:05.222009 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.195:34711 every 8 connection(s)
I20260923 18:10:05.222857 6221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:05.228834 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6221
I20260923 18:10:05.229024 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:05.229314 18611 external_mini_cluster.cc:1428] Setting key 922a074c8bcfbf09311202007cde275f
I20260923 18:10:05.233856 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.225192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:42173 (local address 127.18.44.254:34503)
0923 18:10:05.225407 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.225411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.226571 (+ 1160us) server_negotiation.cc:407] Connection header received
0923 18:10:05.228106 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.228110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.228178 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.228255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:05.229375 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.229973 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.231578 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.231898 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.231993 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.233274 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:05.233277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:05.233279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:05.233299 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:05.233321 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:05.233330 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:05.233403 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.233592 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.233724 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 18:10:05.234797 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.225521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:34503 (local address 127.18.44.195:42173)
0923 18:10:05.226358 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.226418 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.227819 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.228480 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.228489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.228591 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:05.229210 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.229228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.230109 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.230114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.231437 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.231450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.232082 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.232085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.232257 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.232922 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:10:05.232924 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:05.232973 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:10:05.233054 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:05.233449 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:05.233460 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:05.233465 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.233591 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.233877 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":140,"threads_started":1}
I20260923 18:10:05.236428 6349 heartbeater.cc:347] Connected to a master server at 127.18.44.254:34503
I20260923 18:10:05.236706 6349 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:05.237389 6349 heartbeater.cc:511] Master 127.18.44.254:34503 requested a full tablet report, sending...
I20260923 18:10:05.238695 5899 ts_manager.cc:194] Registered new tserver with Master: 49a65fb53fe24d68a35c1f3618ed161f (127.18.44.195:34711)
I20260923 18:10:05.239384 5899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.44.195:42173
I20260923 18:10:05.245493 18611 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:05.252904 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.247969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.18.44.254:34503)
0923 18:10:05.248180 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.248186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.248292 (+ 106us) server_negotiation.cc:407] Connection header received
0923 18:10:05.248362 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.248364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.248410 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.248482 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:05.248872 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.249661 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.250867 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.251096 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.251197 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.252367 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:05.252370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:05.252372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:05.252404 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:05.252427 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:05.252435 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:05.252506 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.252653 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.252735 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:10:05.257174 5899 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:05.260888 5899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:05.279145 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.272228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.18.44.193:44567)
0923 18:10:05.273429 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.273436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.273829 (+ 393us) server_negotiation.cc:407] Connection header received
0923 18:10:05.274030 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.274034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.274183 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.274333 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:05.274733 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.275884 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.277686 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278456 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.278745 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.278814 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.278996 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":260,"threads_started":1}
I20260923 18:10:05.279220 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.271731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.195:34711 (local address 127.0.0.1:39342)
0923 18:10:05.272882 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.272945 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.273116 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.273737 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.273743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.273765 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:05.273945 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.273953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.275401 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.275405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.276900 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.276913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278271 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.278540 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.278599 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.278715 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.279038 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260923 18:10:05.279605 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.272890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.18.44.194:36119)
0923 18:10:05.273339 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.273351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.273836 (+ 485us) server_negotiation.cc:407] Connection header received
0923 18:10:05.273923 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.273928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.274114 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.274266 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:05.274537 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.275798 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.277633 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278541 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278672 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.278864 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.279318 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.279429 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":129,"threads_started":1}
I20260923 18:10:05.279690 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.272649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36119 (local address 127.0.0.1:46634)
0923 18:10:05.273605 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.273620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.273732 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.274263 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.274266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.274284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:05.274397 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.274403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.275959 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.275964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.277469 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.277480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278972 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.279060 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.279080 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.279148 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.279216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":193,"thread_start_us":97,"threads_started":1}
I20260923 18:10:05.279102 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.272133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:44567 (local address 127.0.0.1:33786)
0923 18:10:05.273729 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.273743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.273905 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.274477 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.274480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.274500 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:05.274592 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.274599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.276023 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.276026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.277552 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.277565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278580 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.278685 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.278709 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.278774 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.278915 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"mutex_wait_us":554,"thread_start_us":117,"threads_started":1}
I20260923 18:10:05.279865 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.271967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.18.44.195:34711)
0923 18:10:05.272493 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.272514 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.273057 (+ 543us) server_negotiation.cc:407] Connection header received
0923 18:10:05.273293 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.273300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.273540 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.274116 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:05.274303 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.275281 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.277093 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.278075 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.278194 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.278437 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.279520 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.279642 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":186,"threads_started":1}
I20260923 18:10:05.282963 6283 tablet_service.cc:1467] Processing CreateTablet for tablet 4829bf7061704d73b90b88fbee9950c5 (DEFAULT_TABLE table=test-table [id=90ea898392ee4d7a838ef817a2b064de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:05.282956 6150 tablet_service.cc:1467] Processing CreateTablet for tablet 4829bf7061704d73b90b88fbee9950c5 (DEFAULT_TABLE table=test-table [id=90ea898392ee4d7a838ef817a2b064de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:05.282960 6021 tablet_service.cc:1467] Processing CreateTablet for tablet 4829bf7061704d73b90b88fbee9950c5 (DEFAULT_TABLE table=test-table [id=90ea898392ee4d7a838ef817a2b064de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:05.284273 6150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4829bf7061704d73b90b88fbee9950c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:05.284278 6021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4829bf7061704d73b90b88fbee9950c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:05.284286 6283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4829bf7061704d73b90b88fbee9950c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:05.292273 6366 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Bootstrap starting.
I20260923 18:10:05.293226 6368 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: Bootstrap starting.
I20260923 18:10:05.294143 6367 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Bootstrap starting.
I20260923 18:10:05.295495 6366 tablet_bootstrap.cc:654] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:05.296610 6367 tablet_bootstrap.cc:654] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:05.296829 6366 log.cc:824] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:05.297188 6368 tablet_bootstrap.cc:654] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:05.297689 6367 log.cc:824] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:05.298292 6368 log.cc:824] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:05.299821 6366 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: No bootstrap required, opened a new log
I20260923 18:10:05.299960 6367 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: No bootstrap required, opened a new log
I20260923 18:10:05.300086 6366 ts_tablet_manager.cc:1397] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260923 18:10:05.300146 6368 tablet_bootstrap.cc:492] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: No bootstrap required, opened a new log
I20260923 18:10:05.300226 6367 ts_tablet_manager.cc:1397] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:05.300354 6368 ts_tablet_manager.cc:1397] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 18:10:05.304649 6367 raft_consensus.cc:348] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.304649 6366 raft_consensus.cc:348] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.304768 6368 raft_consensus.cc:348] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.305018 6367 raft_consensus.cc:374] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:05.305018 6366 raft_consensus.cc:374] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:05.305061 6368 raft_consensus.cc:374] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:05.305078 6367 raft_consensus.cc:729] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a65fb53fe24d68a35c1f3618ed161f, State: Initialized, Role: FOLLOWER
I20260923 18:10:05.305081 6366 raft_consensus.cc:729] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5c064753af4792a940437b2ab7ff8a, State: Initialized, Role: FOLLOWER
I20260923 18:10:05.305127 6368 raft_consensus.cc:729] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e3ebb5a59142d49688dec399d23cf6, State: Initialized, Role: FOLLOWER
I20260923 18:10:05.305619 6368 consensus_queue.cc:260] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.305630 6366 consensus_queue.cc:260] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.305630 6367 consensus_queue.cc:260] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.306694 6368 ts_tablet_manager.cc:1428] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:10:05.306730 6367 ts_tablet_manager.cc:1428] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:05.306721 6366 ts_tablet_manager.cc:1428] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 18:10:05.307080 6087 heartbeater.cc:503] Master 127.18.44.254:34503 was elected leader, sending a full tablet report...
I20260923 18:10:05.307078 6218 heartbeater.cc:503] Master 127.18.44.254:34503 was elected leader, sending a full tablet report...
I20260923 18:10:05.307078 6349 heartbeater.cc:503] Master 127.18.44.254:34503 was elected leader, sending a full tablet report...
I20260923 18:10:05.333122 6374 raft_consensus.cc:482] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:05.333349 6374 raft_consensus.cc:504] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.334568 6374 leader_election.cc:288] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5c064753af4792a940437b2ab7ff8a (127.18.44.194:36119), d0e3ebb5a59142d49688dec399d23cf6 (127.18.44.193:44567)
I20260923 18:10:05.339159 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.334899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36119 (local address 127.18.44.195:50881)
0923 18:10:05.335130 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.335148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.335435 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.335944 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.335948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.335967 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:05.336095 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.336103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.337077 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.337081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.338018 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.338028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.338678 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.338682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.338842 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.338860 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.338928 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.339008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":45}
I20260923 18:10:05.339159 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.335092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:50881 (local address 127.18.44.194:36119)
0923 18:10:05.335276 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.335282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.335335 (+ 53us) server_negotiation.cc:407] Connection header received
0923 18:10:05.335617 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.335621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.335687 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.335935 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:05.336220 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.336960 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.338147 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.338560 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.338672 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.338725 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.338963 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.339038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:10:05.340039 6172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4829bf7061704d73b90b88fbee9950c5" candidate_uuid: "49a65fb53fe24d68a35c1f3618ed161f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5c064753af4792a940437b2ab7ff8a" is_pre_election: true
I20260923 18:10:05.340381 6172 raft_consensus.cc:2454] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a65fb53fe24d68a35c1f3618ed161f in term 0.
I20260923 18:10:05.340516 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.334899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:44567 (local address 127.18.44.195:60095)
0923 18:10:05.335699 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.335711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:05.335796 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:05.336024 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:05.336027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:05.336042 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:05.336581 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.336590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.337847 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.337851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.338986 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:05.338996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.339977 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.339981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:05.340107 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.340129 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:10:05.340209 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:10:05.340294 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":343,"thread_start_us":214,"threads_started":1}
I20260923 18:10:05.340579 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.335152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.44.195:60095 (local address 127.18.44.193:44567)
0923 18:10:05.335427 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.335432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.335787 (+ 355us) server_negotiation.cc:407] Connection header received
0923 18:10:05.335884 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.335887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.335938 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.336001 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:05.336841 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.337712 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.339139 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.339686 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.339976 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.340035 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.340341 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.340430 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260923 18:10:05.341073 6237 leader_election.cc:302] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5c064753af4792a940437b2ab7ff8a, 49a65fb53fe24d68a35c1f3618ed161f; no voters:
I20260923 18:10:05.341503 6374 raft_consensus.cc:2781] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:05.341446 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4829bf7061704d73b90b88fbee9950c5" candidate_uuid: "49a65fb53fe24d68a35c1f3618ed161f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e3ebb5a59142d49688dec399d23cf6" is_pre_election: true
I20260923 18:10:05.341605 6374 raft_consensus.cc:482] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:05.341653 6374 raft_consensus.cc:3037] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:05.341794 6041 raft_consensus.cc:2454] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49a65fb53fe24d68a35c1f3618ed161f in term 0.
I20260923 18:10:05.342865 6374 raft_consensus.cc:504] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.343604 6374 leader_election.cc:288] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [CANDIDATE]: Term 1 election: Requested vote from peers 3a5c064753af4792a940437b2ab7ff8a (127.18.44.194:36119), d0e3ebb5a59142d49688dec399d23cf6 (127.18.44.193:44567)
I20260923 18:10:05.344053 6172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4829bf7061704d73b90b88fbee9950c5" candidate_uuid: "49a65fb53fe24d68a35c1f3618ed161f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5c064753af4792a940437b2ab7ff8a"
I20260923 18:10:05.344189 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4829bf7061704d73b90b88fbee9950c5" candidate_uuid: "49a65fb53fe24d68a35c1f3618ed161f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e3ebb5a59142d49688dec399d23cf6"
I20260923 18:10:05.344453 6172 raft_consensus.cc:3037] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:05.344460 6041 raft_consensus.cc:3037] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:05.345857 6041 raft_consensus.cc:2454] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a65fb53fe24d68a35c1f3618ed161f in term 1.
I20260923 18:10:05.345921 6172 raft_consensus.cc:2454] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a65fb53fe24d68a35c1f3618ed161f in term 1.
I20260923 18:10:05.346401 6238 leader_election.cc:302] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a5c064753af4792a940437b2ab7ff8a, 49a65fb53fe24d68a35c1f3618ed161f, d0e3ebb5a59142d49688dec399d23cf6; no voters:
I20260923 18:10:05.346618 6374 raft_consensus.cc:2781] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:05.346972 6374 raft_consensus.cc:686] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 1 LEADER]: Becoming Leader. State: Replica: 49a65fb53fe24d68a35c1f3618ed161f, State: Running, Role: LEADER
I20260923 18:10:05.347462 6374 consensus_queue.cc:237] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } }
I20260923 18:10:05.351148 5897 catalog_manager.cc:5495] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49a65fb53fe24d68a35c1f3618ed161f (127.18.44.195). New cstate: current_term: 1 leader_uuid: "49a65fb53fe24d68a35c1f3618ed161f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a65fb53fe24d68a35c1f3618ed161f" member_type: VOTER last_known_addr { host: "127.18.44.195" port: 34711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:05.384115 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.379816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.18.44.195:34711)
0923 18:10:05.379967 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:05.379970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:05.379985 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:05.380132 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:05.380134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:05.380193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:05.380304 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:05.380553 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.381149 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.382112 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:05.382366 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:05.382466 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:05.382963 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:05.383634 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:05.383849 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:05.383924 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 18:10:05.383990 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:10:05.391222 6172 raft_consensus.cc:1261] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 1 FOLLOWER]: Refusing update from remote peer 49a65fb53fe24d68a35c1f3618ed161f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:05.391222 6041 raft_consensus.cc:1261] T 4829bf7061704d73b90b88fbee9950c5 P d0e3ebb5a59142d49688dec399d23cf6 [term 1 FOLLOWER]: Refusing update from remote peer 49a65fb53fe24d68a35c1f3618ed161f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:05.392063 6374 consensus_queue.cc:1035] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5c064753af4792a940437b2ab7ff8a" member_type: VOTER last_known_addr { host: "127.18.44.194" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:05.392339 6376 consensus_queue.cc:1035] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3ebb5a59142d49688dec399d23cf6" member_type: VOTER last_known_addr { host: "127.18.44.193" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:05.415872 5897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:05.416102 5897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:05.419737 5897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd49696c76274f2e9d9bf19bb5186897: Sending DeleteTablet for 3 replicas of tablet 4829bf7061704d73b90b88fbee9950c5
I20260923 18:10:05.420835 6283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4829bf7061704d73b90b88fbee9950c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:05 UTC) from {username='slave'} at 127.0.0.1:39342
I20260923 18:10:05.421094 6021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4829bf7061704d73b90b88fbee9950c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:05 UTC) from {username='slave'} at 127.0.0.1:33786
I20260923 18:10:05.421164 6150 tablet_service.cc:1514] Processing DeleteTablet for tablet 4829bf7061704d73b90b88fbee9950c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:05 UTC) from {username='slave'} at 127.0.0.1:46634
I20260923 18:10:05.421671 6389 tablet_replica.cc:330] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: stopping tablet replica
I20260923 18:10:05.421934 6391 tablet_replica.cc:330] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: stopping tablet replica
I20260923 18:10:05.422076 6389 raft_consensus.cc:2229] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:05.422133 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5959
I20260923 18:10:05.422788 6389 raft_consensus.cc:2258] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:05.423928 6391 raft_consensus.cc:2229] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:05.424413 6391 raft_consensus.cc:2258] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:05.426584 6391 ts_tablet_manager.cc:1905] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:05.428400 6389 ts_tablet_manager.cc:1905] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:05.430148 6391 ts_tablet_manager.cc:1918] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:05.430289 6391 log.cc:1196] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-1/wal/wals/4829bf7061704d73b90b88fbee9950c5
I20260923 18:10:05.430613 6391 ts_tablet_manager.cc:1939] T 4829bf7061704d73b90b88fbee9950c5 P 3a5c064753af4792a940437b2ab7ff8a: Deleting consensus metadata
W20260923 18:10:05.431325 5885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.44.193:44567 (error 108)
I20260923 18:10:05.432144 6389 ts_tablet_manager.cc:1918] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:05.432718 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.432270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.193:44567 (local address 127.0.0.1:33790)
0923 18:10:05.432503 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.432602 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 18:10:05.432737 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6090
W20260923 18:10:05.433034 5885 catalog_manager.cc:4584] TS d0e3ebb5a59142d49688dec399d23cf6 (127.18.44.193:44567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4829bf7061704d73b90b88fbee9950c5: Network error: Client connection negotiation failed: client connection to 127.18.44.193:44567: connect: Connection refused (error 111)
I20260923 18:10:05.433382 6389 log.cc:1196] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Deleting WAL directory at /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790186727112038-18611-0/minicluster-data/ts-2/wal/wals/4829bf7061704d73b90b88fbee9950c5
I20260923 18:10:05.433815 6389 ts_tablet_manager.cc:1939] T 4829bf7061704d73b90b88fbee9950c5 P 49a65fb53fe24d68a35c1f3618ed161f: Deleting consensus metadata
I20260923 18:10:05.435432 5884 catalog_manager.cc:4841] TS 49a65fb53fe24d68a35c1f3618ed161f (127.18.44.195:34711): tablet 4829bf7061704d73b90b88fbee9950c5 (table test-table [id=90ea898392ee4d7a838ef817a2b064de]) successfully deleted
I20260923 18:10:05.439615 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6221
I20260923 18:10:05.439785 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:05.439440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.194:36119 (local address 127.0.0.1:46638)
0923 18:10:05.439598 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:05.439688 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.194:36119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 18:10:05.445966 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 5868
2026-09-23T18:10:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1308 ms)
[----------] 6 tests from EncryptionPolicyTest (12277 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:05 dist-test-slave-rvjj krb5kdc[6395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:05 dist-test-slave-rvjj krb5kdc[6395](info): set up 2 sockets
Sep 23 18:10:05 dist-test-slave-rvjj krb5kdc[6395](info): commencing operation
krb5kdc: starting...
W20260923 18:10:06.468561 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:06 dist-test-slave-rvjj krb5kdc[6395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:06.604508 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:06.747185 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:06.747574 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:06.747641 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:06.753191 6408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:06.753322 6408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:06.753357 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:06.753381 6408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:06.753401 6408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:06.760509 6408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:06.761994 6408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:06.763190 6408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:06.774632 6414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:06.775224 6416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:06.775714 6408 server_base.cc:1034] running on GCE node
W20260923 18:10:06.775826 6413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:06.776573 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:06.777940 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:06.779173 6408 hybrid_clock.cc:648] HybridClock initialized: now 1790187006779138 us; error 58 us; skew 500 ppm
Sep 23 18:10:06 dist-test-slave-rvjj krb5kdc[6395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:06.783773 6408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:06.785284 6408 webserver.cc:460] Webserver started at http://127.0.0.1:43433/ using document root <none> and password file <none>
I20260923 18:10:06.786093 6408 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:06.786197 6408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:06.786448 6408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:06.789901 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "1c72f1318814478c8c4c390d23a6a197"
format_stamp: "Formatted at 2026-09-23 18:10:06 on dist-test-slave-rvjj"
server_key: "b8a43f10029f5cc99f07c91826669272"
server_key_iv: "5a12f4ffdf5bde8b136c9f1ff237dff1"
server_key_version: "encryptionkey@0"
I20260923 18:10:06.790971 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "1c72f1318814478c8c4c390d23a6a197"
format_stamp: "Formatted at 2026-09-23 18:10:06 on dist-test-slave-rvjj"
server_key: "b8a43f10029f5cc99f07c91826669272"
server_key_iv: "5a12f4ffdf5bde8b136c9f1ff237dff1"
server_key_version: "encryptionkey@0"
I20260923 18:10:06.799656 6408 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260923 18:10:06.804934 6423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:06.807773 6408 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260923 18:10:06.808477 6408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "1c72f1318814478c8c4c390d23a6a197"
format_stamp: "Formatted at 2026-09-23 18:10:06 on dist-test-slave-rvjj"
server_key: "b8a43f10029f5cc99f07c91826669272"
server_key_iv: "5a12f4ffdf5bde8b136c9f1ff237dff1"
server_key_version: "encryptionkey@0"
I20260923 18:10:06.808756 6408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:06.828634 6408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:06.836023 6408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:06.837100 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:06.849036 6408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40893
I20260923 18:10:06.849072 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40893 every 8 connection(s)
I20260923 18:10:06.850519 6408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:06.854234 6476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:06.855593 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6408
I20260923 18:10:06.855747 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:06.856051 18611 external_mini_cluster.cc:1428] Setting key 928e153a28b576e3b52de3320c4cb858
I20260923 18:10:06.862124 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: Bootstrap starting.
I20260923 18:10:06.865350 6476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:06.866293 6476 log.cc:824] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:06 dist-test-slave-rvjj krb5kdc[6395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:10:06.868623 6476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: No bootstrap required, opened a new log
I20260923 18:10:06.874009 6476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260923 18:10:06.874447 6476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:06.874539 6476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72f1318814478c8c4c390d23a6a197, State: Initialized, Role: FOLLOWER
I20260923 18:10:06.875689 6476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260923 18:10:06.875922 6476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:06.876013 6476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:06.876125 6476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:06.877514 6476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260923 18:10:06.878006 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:06.857953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37393 (local address 127.0.0.1:40893)
0923 18:10:06.858636 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:06.858659 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:06.858720 (+ 61us) server_negotiation.cc:407] Connection header received
0923 18:10:06.859833 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:06.859863 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:06.860496 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:06.860990 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:06.862144 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:06.863472 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:06.864518 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:06.865156 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:06.865352 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:06.869084 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:06.869093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:06.869112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:06.869155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:06.871916 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:06.873476 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:06.873485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:06.873493 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:06.873693 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:06.874391 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:06.874395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:06.874397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:06.875469 (+ 1072us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:06.875881 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:06.876503 (+ 622us) server_negotiation.cc:299] Negotiation successful
0923 18:10:06.876796 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":177,"threads_started":1}
I20260923 18:10:06.877960 6476 leader_election.cc:302] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c72f1318814478c8c4c390d23a6a197; no voters:
I20260923 18:10:06.878396 6476 leader_election.cc:288] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:06.878800 6481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:06.879226 6481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72f1318814478c8c4c390d23a6a197, State: Running, Role: LEADER
I20260923 18:10:06.879833 6481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260923 18:10:06.880019 6476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:06.882366 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:06.882612 6482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c72f1318814478c8c4c390d23a6a197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260923 18:10:06.882825 6482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:06.882550 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c72f1318814478c8c4c390d23a6a197. Latest consensus state: current_term: 1 leader_uuid: "1c72f1318814478c8c4c390d23a6a197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c72f1318814478c8c4c390d23a6a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260923 18:10:06.883689 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:06.886526 6495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:06.886691 6495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:06.886844 6496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:06.888149 6496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:06.895447 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:06.885552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54237 (local address 127.0.0.1:40893)
0923 18:10:06.885671 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:06.885676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:06.885951 (+ 275us) server_negotiation.cc:407] Connection header received
0923 18:10:06.886149 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:06.886152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:06.886220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:06.886273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:06.887278 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:06.887890 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:06.888773 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:06.889024 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:06.889117 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:06.890572 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:06.890576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:06.890578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:06.890605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:06.892508 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:06.893468 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:06.893474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:06.893476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:06.893617 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:06.894101 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:06.894105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:06.894107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:06.894399 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:06.894587 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:06.895033 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 18:10:06.895200 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:10:06.896049 6496 catalog_manager.cc:1347] Generated new cluster ID: 18b9946cd20d47209b3bcdb9cdc81fe3
I20260923 18:10:06.896126 6496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:06.922024 6496 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:06.923064 6496 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:06.930369 6496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c72f1318814478c8c4c390d23a6a197: Generated new TSK 0
I20260923 18:10:06.931170 6496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:06.939466 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6408
W20260923 18:10:06.944955 6393 connection.cc:535] client connection to 127.0.0.1:40893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40893): Connection reset by peer (error 104)
2026-09-23T18:10:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:07 dist-test-slave-rvjj krb5kdc[6507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:07 dist-test-slave-rvjj krb5kdc[6507](info): set up 2 sockets
Sep 23 18:10:07 dist-test-slave-rvjj krb5kdc[6507](info): commencing operation
krb5kdc: starting...
W20260923 18:10:07.980620 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:08 dist-test-slave-rvjj krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:08.135510 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:08.271713 6520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:08.272047 6520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:08.272107 6520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:08.276537 6520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:08.276635 6520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:08.276667 6520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:08.276690 6520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:08.276711 6520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:08.282467 6520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:08.283700 6520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:08.284751 6520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:08.294801 6528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:08.294793 6526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:08.294871 6520 server_base.cc:1034] running on GCE node
W20260923 18:10:08.294801 6525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:08.295876 6520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:08.297462 6520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:08.298830 6520 hybrid_clock.cc:648] HybridClock initialized: now 1790187008298781 us; error 82 us; skew 500 ppm
Sep 23 18:10:08 dist-test-slave-rvjj krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:08.302920 6520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:08.304162 6520 webserver.cc:460] Webserver started at http://127.0.0.1:36139/ using document root <none> and password file <none>
I20260923 18:10:08.304802 6520 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:08.304878 6520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:08.305094 6520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:08.307512 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "96e752ec493348148596b1c0c7cd7347"
format_stamp: "Formatted at 2026-09-23 18:10:08 on dist-test-slave-rvjj"
server_key: "d7142abe0c6e9443caa3b3d726b6912b"
server_key_iv: "7589d6b8e331a78fd95b7ed86be47ee2"
server_key_version: "encryptionkey@0"
I20260923 18:10:08.308189 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "96e752ec493348148596b1c0c7cd7347"
format_stamp: "Formatted at 2026-09-23 18:10:08 on dist-test-slave-rvjj"
server_key: "d7142abe0c6e9443caa3b3d726b6912b"
server_key_iv: "7589d6b8e331a78fd95b7ed86be47ee2"
server_key_version: "encryptionkey@0"
I20260923 18:10:08.312741 6520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 18:10:08.315816 6535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:08.317353 6520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:10:08.317504 6520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "96e752ec493348148596b1c0c7cd7347"
format_stamp: "Formatted at 2026-09-23 18:10:08 on dist-test-slave-rvjj"
server_key: "d7142abe0c6e9443caa3b3d726b6912b"
server_key_iv: "7589d6b8e331a78fd95b7ed86be47ee2"
server_key_version: "encryptionkey@0"
I20260923 18:10:08.317622 6520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:08.339555 6520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:08.344633 6520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:08.344856 6520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:08.353539 6520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42447
I20260923 18:10:08.353567 6587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42447 every 8 connection(s)
I20260923 18:10:08.354701 6520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:08.358184 6588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:08.364234 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6520
I20260923 18:10:08.364423 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:08.364712 18611 external_mini_cluster.cc:1428] Setting key fd3e00942644be69e08999fd0c9cbb01
I20260923 18:10:08.365497 6588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: Bootstrap starting.
I20260923 18:10:08.368229 6588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:08.369182 6588 log.cc:824] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:08.371711 6588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: No bootstrap required, opened a new log
I20260923 18:10:08.376322 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:08.366206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47959 (local address 127.0.0.1:42447)
0923 18:10:08.366689 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:08.366706 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:08.366747 (+ 41us) server_negotiation.cc:407] Connection header received
0923 18:10:08.367557 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:08.367579 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:08.367989 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:08.368312 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:08.368760 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:08.370247 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:08.371721 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:08.372402 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:08.372557 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:08.373695 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:08.373701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:08.373717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:08.374340 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:08.374468 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:08.374499 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:08.374630 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:08.375004 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 18:10:08.375194 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":142,"threads_started":1}
I20260923 18:10:08.376468 6588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260923 18:10:08.376816 6588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:08.376883 6588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e752ec493348148596b1c0c7cd7347, State: Initialized, Role: FOLLOWER
I20260923 18:10:08.377315 6588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260923 18:10:08.377462 6588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:08.377532 6588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:08.377632 6588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:08.378710 6588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260923 18:10:08.379240 6588 leader_election.cc:302] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e752ec493348148596b1c0c7cd7347; no voters:
I20260923 18:10:08.379621 6588 leader_election.cc:288] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:08.379737 6593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:08.380087 6593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [term 1 LEADER]: Becoming Leader. State: Replica: 96e752ec493348148596b1c0c7cd7347, State: Running, Role: LEADER
I20260923 18:10:08.380100 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:08.380707 6593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260923 18:10:08.381225 6588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:08.382913 6594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96e752ec493348148596b1c0c7cd7347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } } }
I20260923 18:10:08.383116 6594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:08.383546 6594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e752ec493348148596b1c0c7cd7347. Latest consensus state: current_term: 1 leader_uuid: "96e752ec493348148596b1c0c7cd7347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e752ec493348148596b1c0c7cd7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } } }
I20260923 18:10:08.383944 6594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:08.387923 6612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:08.388062 6612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:08.388132 6611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:08.389173 6611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:08.391553 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:08.385112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50609 (local address 127.0.0.1:42447)
0923 18:10:08.385262 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:08.385266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:08.385823 (+ 557us) server_negotiation.cc:407] Connection header received
0923 18:10:08.386032 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:08.386035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:08.386087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:08.386185 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:08.386622 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:08.387686 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:08.388930 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:08.389390 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:08.389525 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:08.390777 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:08.390782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:08.390785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:08.390803 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:08.390840 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:08.390910 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:08.391039 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:08.391213 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 18:10:08.391394 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:10:08.396282 6611 catalog_manager.cc:1347] Generated new cluster ID: de9189b75b114d0d92d04b51016d0892
I20260923 18:10:08.396391 6611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:08.406376 6611 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:08.407536 6611 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:08.413802 6611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96e752ec493348148596b1c0c7cd7347: Generated new TSK 0
I20260923 18:10:08.414669 6611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:08.422134 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6520
W20260923 18:10:08.427536 6505 connection.cc:535] client connection to 127.0.0.1:42447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42447): Connection reset by peer (error 104)
2026-09-23T18:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:08 dist-test-slave-rvjj krb5kdc[6619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:08 dist-test-slave-rvjj krb5kdc[6619](info): set up 2 sockets
Sep 23 18:10:08 dist-test-slave-rvjj krb5kdc[6619](info): commencing operation
krb5kdc: starting...
W20260923 18:10:09.432680 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:09 dist-test-slave-rvjj krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:09.572706 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:09.691121 6632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:09.691521 6632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:09.691591 6632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:09.696293 6632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:09.696410 6632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:09.696455 6632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:09.696476 6632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:09.696497 6632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:09.702490 6632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:09.703743 6632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:09.704811 6632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:10:09.714367 6632 server_base.cc:1034] running on GCE node
W20260923 18:10:09.714376 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:09.714401 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:09.714376 6637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:09.715473 6632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:09.716719 6632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:09.717931 6632 hybrid_clock.cc:648] HybridClock initialized: now 1790187009717872 us; error 78 us; skew 500 ppm
Sep 23 18:10:09 dist-test-slave-rvjj krb5kdc[6619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:09.721472 6632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:09.722682 6632 webserver.cc:460] Webserver started at http://127.0.0.1:41309/ using document root <none> and password file <none>
I20260923 18:10:09.723538 6632 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:09.723656 6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:09.723924 6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:09.726255 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "3882e13c22804c648abdf75a63feb5f1"
format_stamp: "Formatted at 2026-09-23 18:10:09 on dist-test-slave-rvjj"
server_key: "79e6a49e7992ab01bfbf82f64d393f21"
server_key_iv: "c7beb47575e4f2f9204aa45e4c3866bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:09.726904 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "3882e13c22804c648abdf75a63feb5f1"
format_stamp: "Formatted at 2026-09-23 18:10:09 on dist-test-slave-rvjj"
server_key: "79e6a49e7992ab01bfbf82f64d393f21"
server_key_iv: "c7beb47575e4f2f9204aa45e4c3866bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:09.731244 6632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:10:09.734306 6647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:09.735898 6632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:09.736053 6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "3882e13c22804c648abdf75a63feb5f1"
format_stamp: "Formatted at 2026-09-23 18:10:09 on dist-test-slave-rvjj"
server_key: "79e6a49e7992ab01bfbf82f64d393f21"
server_key_iv: "c7beb47575e4f2f9204aa45e4c3866bf"
server_key_version: "encryptionkey@0"
I20260923 18:10:09.736182 6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:09.741894 6632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:09.745777 6632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:09.746031 6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:09.753497 6632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44129
I20260923 18:10:09.753563 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44129 every 8 connection(s)
I20260923 18:10:09.754505 6632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:09.757951 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:09.760221 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6632
I20260923 18:10:09.760383 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:09.760742 18611 external_mini_cluster.cc:1428] Setting key 53cc8eb453b8812b9595a8dc6713150b
I20260923 18:10:09.765447 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1: Bootstrap starting.
I20260923 18:10:09.766390 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:09.762279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32921 (local address 127.0.0.1:44129)
0923 18:10:09.762848 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:09.762866 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:09.762915 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:10:09.763681 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:09.763702 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:09.764016 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:09.764433 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:09.765146 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:09.765150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:09.765167 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:09.765179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:09.765234 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:09.765255 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:09.765396 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:09.765476 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 18:10:09.765646 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":187,"threads_started":1}
I20260923 18:10:09.767902 6700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:09.768719 6700 log.cc:824] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:09.769318 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:09.770857 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1: No bootstrap required, opened a new log
I20260923 18:10:09.773365 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:09.771478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40027 (local address 127.0.0.1:44129)
0923 18:10:09.771599 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:09.771604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:09.771876 (+ 272us) server_negotiation.cc:407] Connection header received
0923 18:10:09.771960 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:09.771962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:09.771998 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:09.772093 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:09.772938 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:09.772942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:09.772945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:09.772962 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:09.772994 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:09.773005 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:09.773100 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:09.773192 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 18:10:09.773230 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:10:09.775295 6700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260923 18:10:09.775596 6700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:09.775633 6700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3882e13c22804c648abdf75a63feb5f1, State: Initialized, Role: FOLLOWER
I20260923 18:10:09.776093 6700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260923 18:10:09.776235 6700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:09.776301 6700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:09.776400 6700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:09.777418 6700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260923 18:10:09.777778 6700 leader_election.cc:302] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3882e13c22804c648abdf75a63feb5f1; no voters:
I20260923 18:10:09.778110 6700 leader_election.cc:288] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:09.778299 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:09.778568 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3882e13c22804c648abdf75a63feb5f1, State: Running, Role: LEADER
I20260923 18:10:09.778952 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260923 18:10:09.779440 6700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:09.780633 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3882e13c22804c648abdf75a63feb5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260923 18:10:09.780767 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:09.781229 6719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:09.781499 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3882e13c22804c648abdf75a63feb5f1. Latest consensus state: current_term: 1 leader_uuid: "3882e13c22804c648abdf75a63feb5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3882e13c22804c648abdf75a63feb5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260923 18:10:09.782086 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:09.784703 6719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:09.792070 6719 catalog_manager.cc:1347] Generated new cluster ID: 0b2c5a27995e442c85b3497bdbd655b3
I20260923 18:10:09.792184 6719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:09.816167 6719 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:09.816562 6719 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:09.826102 6719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3882e13c22804c648abdf75a63feb5f1: Generated new TSK 0
I20260923 18:10:09.827069 6719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:09.834419 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6632
2026-09-23T18:10:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:09 dist-test-slave-rvjj krb5kdc[6731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:09 dist-test-slave-rvjj krb5kdc[6731](info): set up 2 sockets
Sep 23 18:10:09 dist-test-slave-rvjj krb5kdc[6731](info): commencing operation
krb5kdc: starting...
W20260923 18:10:10.840799 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:10 dist-test-slave-rvjj krb5kdc[6731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:11.000648 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:11.161747 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:11.162323 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:11.162436 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:11.169592 6744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:11.169734 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:11.169766 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:11.169791 6744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:11.169814 6744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:11.177838 6744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:11.179577 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:11.181010 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:10:11.192363 6744 server_base.cc:1034] running on GCE node
W20260923 18:10:11.192245 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:11.192498 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:11.192240 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:11.193413 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:11.195163 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:11.196410 6744 hybrid_clock.cc:648] HybridClock initialized: now 1790187011196379 us; error 59 us; skew 500 ppm
Sep 23 18:10:11 dist-test-slave-rvjj krb5kdc[6731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:11.205935 6744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:11.207656 6744 webserver.cc:460] Webserver started at http://127.0.0.1:42403/ using document root <none> and password file <none>
I20260923 18:10:11.208441 6744 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:11.208520 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:11.208751 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:11.211586 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "1ac2260735f54b408db28916c24e7442"
format_stamp: "Formatted at 2026-09-23 18:10:11 on dist-test-slave-rvjj"
server_key: "436bc7826a7212231e048563e3c02b69"
server_key_iv: "25ba637d82912d7ed1757f3787162e9c"
server_key_version: "encryptionkey@0"
I20260923 18:10:11.212327 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "1ac2260735f54b408db28916c24e7442"
format_stamp: "Formatted at 2026-09-23 18:10:11 on dist-test-slave-rvjj"
server_key: "436bc7826a7212231e048563e3c02b69"
server_key_iv: "25ba637d82912d7ed1757f3787162e9c"
server_key_version: "encryptionkey@0"
I20260923 18:10:11.217693 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 18:10:11.221539 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:11.223865 6744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:11.224047 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "1ac2260735f54b408db28916c24e7442"
format_stamp: "Formatted at 2026-09-23 18:10:11 on dist-test-slave-rvjj"
server_key: "436bc7826a7212231e048563e3c02b69"
server_key_iv: "25ba637d82912d7ed1757f3787162e9c"
server_key_version: "encryptionkey@0"
I20260923 18:10:11.224431 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:11.252496 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:11.256767 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:11.256991 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:11.266430 6744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37361
I20260923 18:10:11.266475 6811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37361 every 8 connection(s)
I20260923 18:10:11.267802 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:11.270974 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6744
I20260923 18:10:11.271136 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:11.271242 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:11.271495 18611 external_mini_cluster.cc:1428] Setting key 6941eda840583809342eaf49c9ea0143
I20260923 18:10:11.279750 6812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: Bootstrap starting.
I20260923 18:10:11.282804 6812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:11.283850 6812 log.cc:824] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:11 dist-test-slave-rvjj krb5kdc[6731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:10:11.286242 6812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: No bootstrap required, opened a new log
I20260923 18:10:11.290792 6812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260923 18:10:11.291100 6812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:11.291162 6812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac2260735f54b408db28916c24e7442, State: Initialized, Role: FOLLOWER
I20260923 18:10:11.291706 6812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260923 18:10:11.291877 6812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:11.291952 6812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:11.291961 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:11.273582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57747 (local address 127.0.0.1:37361)
0923 18:10:11.274177 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:11.274204 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:11.274264 (+ 60us) server_negotiation.cc:407] Connection header received
0923 18:10:11.275218 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:11.275246 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:11.275848 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:11.276267 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:11.277476 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:11.279026 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:11.280571 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:11.281333 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:11.281568 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:11.285334 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:11.285343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:11.285362 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:11.285403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:11.287703 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:11.288517 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:11.288525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:11.288534 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:11.288718 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:11.289152 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:11.289156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:11.289158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:11.289796 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:11.290037 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:11.290467 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 18:10:11.290838 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":182,"threads_started":1}
I20260923 18:10:11.292076 6812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:11.293088 6812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260923 18:10:11.293459 6812 leader_election.cc:302] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ac2260735f54b408db28916c24e7442; no voters:
I20260923 18:10:11.293805 6812 leader_election.cc:288] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:11.294039 6817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:11.294450 6817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [term 1 LEADER]: Becoming Leader. State: Replica: 1ac2260735f54b408db28916c24e7442, State: Running, Role: LEADER
I20260923 18:10:11.294914 6817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260923 18:10:11.295274 6812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:11.297063 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:11.297130 6819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ac2260735f54b408db28916c24e7442. Latest consensus state: current_term: 1 leader_uuid: "1ac2260735f54b408db28916c24e7442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } } }
I20260923 18:10:11.297317 6819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:11.297762 6818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ac2260735f54b408db28916c24e7442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac2260735f54b408db28916c24e7442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } } }
I20260923 18:10:11.297868 6818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:11.302506 6832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:11.302639 6832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:11.302750 6831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:11.303952 6831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:11.312276 6831 catalog_manager.cc:1347] Generated new cluster ID: 5f51f417c8e84815b79a0d2677e771e4
I20260923 18:10:11.312558 6831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:11.313978 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:11.299684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34085 (local address 127.0.0.1:37361)
0923 18:10:11.299866 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:11.299869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:11.303240 (+ 3371us) server_negotiation.cc:407] Connection header received
0923 18:10:11.303462 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:11.303467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:11.303543 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:11.303674 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:11.304798 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:11.305660 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:11.307008 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:11.307340 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:11.307451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:11.309057 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:11.309061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:11.309063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:11.309098 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:11.311465 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:11.312258 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:11.312263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:11.312265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:11.312425 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:11.312867 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:11.312870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:11.312871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:11.313133 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:11.313283 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:11.313618 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 18:10:11.313801 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 18:10:11.332175 6831 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:11.333652 6831 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:11.339774 6831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ac2260735f54b408db28916c24e7442: Generated new TSK 0
I20260923 18:10:11.340862 6831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:11.348618 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6744
2026-09-23T18:10:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1516 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:11 dist-test-slave-rvjj krb5kdc[6843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:11 dist-test-slave-rvjj krb5kdc[6843](info): set up 2 sockets
Sep 23 18:10:11 dist-test-slave-rvjj krb5kdc[6843](info): commencing operation
krb5kdc: starting...
W20260923 18:10:12.352595 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:12 dist-test-slave-rvjj krb5kdc[6843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:12.507413 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:12.666239 6856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:12.666698 6856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:12.666770 6856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:12.671934 6856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:12.672070 6856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:12.672101 6856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:12.672132 6856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:12.672155 6856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:12.681289 6856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:12.683120 6856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:12.684770 6856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:12.698539 6861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:12.698498 6862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:12.698607 6856 server_base.cc:1034] running on GCE node
W20260923 18:10:12.698498 6864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:12.700057 6856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:12.702525 6856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:12.703872 6856 hybrid_clock.cc:648] HybridClock initialized: now 1790187012703806 us; error 67 us; skew 500 ppm
Sep 23 18:10:12 dist-test-slave-rvjj krb5kdc[6843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:12.709383 6856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:12.711534 6856 webserver.cc:460] Webserver started at http://127.0.0.1:43807/ using document root <none> and password file <none>
I20260923 18:10:12.712476 6856 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:12.712581 6856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:12.712852 6856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:12.715917 6856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "0ab68821a7234187aa72d5989fa7bb7b"
format_stamp: "Formatted at 2026-09-23 18:10:12 on dist-test-slave-rvjj"
server_key: "7f9cb7afce0e45a40a93fb10f2539632"
server_key_iv: "11d7a56ef5b9334fa7fda973e1fcbb34"
server_key_version: "encryptionkey@0"
I20260923 18:10:12.716805 6856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "0ab68821a7234187aa72d5989fa7bb7b"
format_stamp: "Formatted at 2026-09-23 18:10:12 on dist-test-slave-rvjj"
server_key: "7f9cb7afce0e45a40a93fb10f2539632"
server_key_iv: "11d7a56ef5b9334fa7fda973e1fcbb34"
server_key_version: "encryptionkey@0"
I20260923 18:10:12.722833 6856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:12.727226 6871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:12.729218 6856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:12.729418 6856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "0ab68821a7234187aa72d5989fa7bb7b"
format_stamp: "Formatted at 2026-09-23 18:10:12 on dist-test-slave-rvjj"
server_key: "7f9cb7afce0e45a40a93fb10f2539632"
server_key_iv: "11d7a56ef5b9334fa7fda973e1fcbb34"
server_key_version: "encryptionkey@0"
I20260923 18:10:12.729589 6856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:12.759368 6856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:12.764153 6856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:12.764405 6856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:12.774760 6856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39569
I20260923 18:10:12.774780 6923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39569 every 8 connection(s)
I20260923 18:10:12.776216 6856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:12.778249 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6856
I20260923 18:10:12.778407 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:12.778775 18611 external_mini_cluster.cc:1428] Setting key 55b69d85e4246f8e20b9d13ad879bc18
I20260923 18:10:12.780174 6924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:12.788702 6924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b: Bootstrap starting.
I20260923 18:10:12.791652 6924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:12.792479 6924 log.cc:824] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:12.793062 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:12.781006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36183 (local address 127.0.0.1:39569)
0923 18:10:12.781825 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:12.781842 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:12.781931 (+ 89us) server_negotiation.cc:407] Connection header received
0923 18:10:12.783201 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:12.783242 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:12.783948 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:12.784439 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:12.784943 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:12.786233 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:12.787786 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:12.788600 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:12.788779 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:12.790089 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:12.790097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:12.790123 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:12.790798 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:12.790910 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:12.790949 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:12.791079 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:12.791361 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 18:10:12.791787 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":296,"threads_started":1}
I20260923 18:10:12.794900 6924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b: No bootstrap required, opened a new log
I20260923 18:10:12.797699 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:12.799336 6924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260923 18:10:12.799635 6924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:12.799681 6924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab68821a7234187aa72d5989fa7bb7b, State: Initialized, Role: FOLLOWER
I20260923 18:10:12.800192 6924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260923 18:10:12.800379 6924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:12.800542 6924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:12.800731 6924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:12.801772 6924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260923 18:10:12.802160 6924 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab68821a7234187aa72d5989fa7bb7b; no voters:
I20260923 18:10:12.802510 6924 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:12.802615 6934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:12.802820 6934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [term 1 LEADER]: Becoming Leader. State: Replica: 0ab68821a7234187aa72d5989fa7bb7b, State: Running, Role: LEADER
I20260923 18:10:12.803332 6934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260923 18:10:12.804103 6924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:12.805361 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab68821a7234187aa72d5989fa7bb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } } }
I20260923 18:10:12.805583 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:12.806211 6943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:12.806550 6936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab68821a7234187aa72d5989fa7bb7b. Latest consensus state: current_term: 1 leader_uuid: "0ab68821a7234187aa72d5989fa7bb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab68821a7234187aa72d5989fa7bb7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } } }
I20260923 18:10:12.807188 6936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:12.807955 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:12.800533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58157 (local address 127.0.0.1:39569)
0923 18:10:12.800711 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:12.800716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:12.800953 (+ 237us) server_negotiation.cc:407] Connection header received
0923 18:10:12.801144 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:12.801148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:12.801211 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:12.801290 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:12.801635 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:12.802398 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:12.803689 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:12.804007 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:12.804129 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:12.807314 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:12.807319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:12.807323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:12.807346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:12.807396 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:12.807416 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:12.807477 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:12.807735 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 18:10:12.807824 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:10:12.810822 6943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:12.819183 6943 catalog_manager.cc:1347] Generated new cluster ID: 17ce3aa8a0cc49e194b016855b3108e1
I20260923 18:10:12.819319 6943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:12.852815 6943 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:12.854313 6943 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:12.862097 6943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab68821a7234187aa72d5989fa7bb7b: Generated new TSK 0
I20260923 18:10:12.863533 6943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:12.879855 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6856
2026-09-23T18:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:12 dist-test-slave-rvjj krb5kdc[6955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:12 dist-test-slave-rvjj krb5kdc[6955](info): set up 2 sockets
Sep 23 18:10:12 dist-test-slave-rvjj krb5kdc[6955](info): commencing operation
krb5kdc: starting...
W20260923 18:10:13.888616 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:13 dist-test-slave-rvjj krb5kdc[6955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:14.043051 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:14.195161 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:14.195829 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:14.195981 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:14.201861 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:14.201967 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:14.202004 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:14.202029 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:14.202059 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:14.209141 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:14.211759 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:14.213472 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:14.226011 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:14.225920 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:14.226055 6968 server_base.cc:1034] running on GCE node
W20260923 18:10:14.225951 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:14.227171 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:14.228965 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:14.230408 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790187014230348 us; error 84 us; skew 500 ppm
Sep 23 18:10:14 dist-test-slave-rvjj krb5kdc[6955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:14.234793 6968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:14.236294 6968 webserver.cc:460] Webserver started at http://127.0.0.1:44969/ using document root <none> and password file <none>
I20260923 18:10:14.237061 6968 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:14.237142 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:14.237390 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:14.240898 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "737d60db8e684288992c948b28e57dd7"
format_stamp: "Formatted at 2026-09-23 18:10:14 on dist-test-slave-rvjj"
server_key: "28a235a273606b7d7c3f0b89c9e2bf44"
server_key_iv: "2707c72fc75f5ad67b30963565d9a335"
server_key_version: "encryptionkey@0"
I20260923 18:10:14.241829 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "737d60db8e684288992c948b28e57dd7"
format_stamp: "Formatted at 2026-09-23 18:10:14 on dist-test-slave-rvjj"
server_key: "28a235a273606b7d7c3f0b89c9e2bf44"
server_key_iv: "2707c72fc75f5ad67b30963565d9a335"
server_key_version: "encryptionkey@0"
I20260923 18:10:14.248540 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:10:14.254665 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:14.256896 6968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:14.257148 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "737d60db8e684288992c948b28e57dd7"
format_stamp: "Formatted at 2026-09-23 18:10:14 on dist-test-slave-rvjj"
server_key: "28a235a273606b7d7c3f0b89c9e2bf44"
server_key_iv: "2707c72fc75f5ad67b30963565d9a335"
server_key_version: "encryptionkey@0"
I20260923 18:10:14.257328 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:14.264127 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:14.268920 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:14.269186 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:14.279827 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37121
I20260923 18:10:14.279867 7035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37121 every 8 connection(s)
I20260923 18:10:14.281220 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:14.282815 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 6968
I20260923 18:10:14.282979 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:14.283456 18611 external_mini_cluster.cc:1428] Setting key 02881f88594a4157561521a3e3c8956e
I20260923 18:10:14.285058 7036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:14.290637 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:14.285348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35527 (local address 127.0.0.1:37121)
0923 18:10:14.285933 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:14.285954 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:14.286004 (+ 50us) server_negotiation.cc:407] Connection header received
0923 18:10:14.286844 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:14.286880 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:14.287475 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:14.287938 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:14.288925 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:14.288931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:14.288947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:14.288961 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:14.289033 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:14.289130 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:14.289293 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:14.289391 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 18:10:14.289605 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":177,"threads_started":1}
I20260923 18:10:14.292142 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: Bootstrap starting.
I20260923 18:10:14.294698 7036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:14.294989 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:14.295624 7036 log.cc:824] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:14.298627 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: No bootstrap required, opened a new log
I20260923 18:10:14.300551 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:14.298039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34007 (local address 127.0.0.1:37121)
0923 18:10:14.298187 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:14.298193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:14.298513 (+ 320us) server_negotiation.cc:407] Connection header received
0923 18:10:14.298739 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:14.298742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:14.298794 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:14.298874 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:14.300002 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:14.300006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:14.300009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:14.300023 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:14.300063 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:14.300076 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:14.300247 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:14.300332 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 18:10:14.300387 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:10:14.303340 7036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } }
I20260923 18:10:14.303653 7036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:14.303695 7036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737d60db8e684288992c948b28e57dd7, State: Initialized, Role: FOLLOWER
I20260923 18:10:14.304145 7036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } }
I20260923 18:10:14.304298 7036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:14.304355 7036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:14.304468 7036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:14.305575 7036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } }
I20260923 18:10:14.305971 7036 leader_election.cc:302] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737d60db8e684288992c948b28e57dd7; no voters:
I20260923 18:10:14.306336 7036 leader_election.cc:288] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:14.306859 7046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:14.307328 7046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 737d60db8e684288992c948b28e57dd7, State: Running, Role: LEADER
I20260923 18:10:14.307898 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } }
I20260923 18:10:14.308079 7036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:14.310702 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737d60db8e684288992c948b28e57dd7. Latest consensus state: current_term: 1 leader_uuid: "737d60db8e684288992c948b28e57dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } } }
I20260923 18:10:14.310732 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737d60db8e684288992c948b28e57dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d60db8e684288992c948b28e57dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37121 } } }
I20260923 18:10:14.310971 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:14.310968 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:14.314569 7060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:14.314689 7060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:14.314780 7061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:14.315627 7061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:14.323396 7061 catalog_manager.cc:1347] Generated new cluster ID: 1a7bc0d2e5bf4fbaa250b46fc7cf2113
I20260923 18:10:14.323501 7061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:14.344043 7061 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:14.344467 7061 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:14.353492 7061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737d60db8e684288992c948b28e57dd7: Generated new TSK 0
I20260923 18:10:14.354398 7061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:14.365007 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 6968
2026-09-23T18:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:14 dist-test-slave-rvjj krb5kdc[7067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:14 dist-test-slave-rvjj krb5kdc[7067](info): set up 2 sockets
Sep 23 18:10:14 dist-test-slave-rvjj krb5kdc[7067](info): commencing operation
krb5kdc: starting...
W20260923 18:10:15.436952 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:15.588032 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:15.717490 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:15.717831 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:15.717896 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:15.722900 7080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:15.723049 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:15.723083 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:15.723106 7080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:15.723126 7080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:15.730612 7080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:15.732096 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:15.733289 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:15.743835 7086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:15.743924 7085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:15.743924 7088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:15.744223 7080 server_base.cc:1034] running on GCE node
I20260923 18:10:15.745253 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:15.747025 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:15.748383 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790187015748335 us; error 75 us; skew 500 ppm
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:15.752671 7080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:15.754226 7080 webserver.cc:460] Webserver started at http://127.0.0.1:32987/ using document root <none> and password file <none>
I20260923 18:10:15.755033 7080 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:15.755141 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:15.755455 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:15.758646 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "ec64136c6bd4496190c9366e0716ec9b"
format_stamp: "Formatted at 2026-09-23 18:10:15 on dist-test-slave-rvjj"
server_key: "1dcb41bf702525ca7e0ef5488083fc9c"
server_key_iv: "70435862d235e94b72bc4be093d8cdcb"
server_key_version: "encryptionkey@0"
I20260923 18:10:15.759596 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "ec64136c6bd4496190c9366e0716ec9b"
format_stamp: "Formatted at 2026-09-23 18:10:15 on dist-test-slave-rvjj"
server_key: "1dcb41bf702525ca7e0ef5488083fc9c"
server_key_iv: "70435862d235e94b72bc4be093d8cdcb"
server_key_version: "encryptionkey@0"
I20260923 18:10:15.765434 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:15.769222 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:15.771162 7080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 18:10:15.771456 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "ec64136c6bd4496190c9366e0716ec9b"
format_stamp: "Formatted at 2026-09-23 18:10:15 on dist-test-slave-rvjj"
server_key: "1dcb41bf702525ca7e0ef5488083fc9c"
server_key_iv: "70435862d235e94b72bc4be093d8cdcb"
server_key_version: "encryptionkey@0"
I20260923 18:10:15.771632 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:15.807233 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:15.811641 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:15.811893 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:15.821290 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33435
I20260923 18:10:15.821434 7147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33435 every 8 connection(s)
I20260923 18:10:15.823822 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:15.828054 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:15.828382 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7080
I20260923 18:10:15.828580 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:15.828922 18611 external_mini_cluster.cc:1428] Setting key 37e16b955a0f0fe05424df62aaa9d6b6
I20260923 18:10:15.835314 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b: Bootstrap starting.
I20260923 18:10:15.838218 7148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:15.839224 7148 log.cc:824] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:10:15.842636 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b: No bootstrap required, opened a new log
I20260923 18:10:15.848434 7148 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } }
I20260923 18:10:15.849071 7148 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:15.849182 7148 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec64136c6bd4496190c9366e0716ec9b, State: Initialized, Role: FOLLOWER
I20260923 18:10:15.849287 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:15.830585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:50257 (local address 127.0.0.1:33435)
0923 18:10:15.831083 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:15.831231 (+ 148us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:15.831368 (+ 137us) server_negotiation.cc:407] Connection header received
0923 18:10:15.832382 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:15.832409 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:15.832967 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:15.833391 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:15.834495 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:15.835900 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:15.837366 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:15.838139 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:15.838349 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:15.841856 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:15.841864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:15.841883 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:15.841935 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:15.844191 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:15.845234 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:15.845243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:15.845251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:15.845574 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:15.846389 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:15.846392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:15.846394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:15.846968 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:15.847329 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:15.847812 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 18:10:15.848123 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":141,"threads_started":1}
I20260923 18:10:15.849844 7148 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } }
I20260923 18:10:15.850035 7148 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:15.850198 7148 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:15.850332 7148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:15.851764 7148 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } }
I20260923 18:10:15.852274 7148 leader_election.cc:302] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec64136c6bd4496190c9366e0716ec9b; no voters:
I20260923 18:10:15.852777 7148 leader_election.cc:288] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:15.852962 7153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:15.853466 7153 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [term 1 LEADER]: Becoming Leader. State: Replica: ec64136c6bd4496190c9366e0716ec9b, State: Running, Role: LEADER
I20260923 18:10:15.853662 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:15.853919 7153 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } }
I20260923 18:10:15.854601 7148 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:15.856004 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec64136c6bd4496190c9366e0716ec9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } } }
I20260923 18:10:15.856235 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:15.856349 7155 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec64136c6bd4496190c9366e0716ec9b. Latest consensus state: current_term: 1 leader_uuid: "ec64136c6bd4496190c9366e0716ec9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec64136c6bd4496190c9366e0716ec9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33435 } } }
I20260923 18:10:15.856470 7155 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:15.856762 7166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:15.861240 7166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:15.869272 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:15.857911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:44531 (local address 127.0.0.1:33435)
0923 18:10:15.858076 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:15.858081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:15.858213 (+ 132us) server_negotiation.cc:407] Connection header received
0923 18:10:15.858449 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:15.858454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:15.858519 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:15.858580 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:15.859768 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:15.860686 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:15.862300 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:15.862572 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:15.862674 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:15.864639 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:15.864643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:15.864646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:15.864683 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:15.866624 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:15.867592 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:15.867596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:15.867598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:15.867718 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:15.868200 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:15.868204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:15.868206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:15.868449 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:15.868603 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:15.868961 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 18:10:15.869077 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:10:15.870268 7166 catalog_manager.cc:1347] Generated new cluster ID: 6c944c58c59e42a59074edd74bf7edc2
I20260923 18:10:15.870358 7166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:15.879429 7166 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:15.880774 7166 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:15.894173 7166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec64136c6bd4496190c9366e0716ec9b: Generated new TSK 0
I20260923 18:10:15.895498 7166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:15.901849 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7080
W20260923 18:10:15.907698 7065 connection.cc:535] client connection to 127.0.0.1:33435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33435): Connection reset by peer (error 104)
2026-09-23T18:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7179](info): set up 2 sockets
Sep 23 18:10:15 dist-test-slave-rvjj krb5kdc[7179](info): commencing operation
krb5kdc: starting...
W20260923 18:10:16.921952 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:17 dist-test-slave-rvjj krb5kdc[7179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:17.075505 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:17.225580 7192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:17.226017 7192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:17.226079 7192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:17.231201 7192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:17.231360 7192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:17.231386 7192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:17.231406 7192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:17.231423 7192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:17.238435 7192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:17.240142 7192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:17.241509 7192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:17.253046 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:17.253057 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:17.253377 7192 server_base.cc:1034] running on GCE node
W20260923 18:10:17.253057 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:17.254382 7192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:17.255816 7192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:17.257041 7192 hybrid_clock.cc:648] HybridClock initialized: now 1790187017257028 us; error 44 us; skew 500 ppm
Sep 23 18:10:17 dist-test-slave-rvjj krb5kdc[7179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:17.265523 7192 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:17.266731 7192 webserver.cc:460] Webserver started at http://127.0.0.1:39561/ using document root <none> and password file <none>
I20260923 18:10:17.267498 7192 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:17.267597 7192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:17.267900 7192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:17.270208 7192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "bd2bf8cd10044e5a9d96771205299027"
format_stamp: "Formatted at 2026-09-23 18:10:17 on dist-test-slave-rvjj"
server_key: "7280bc95efa1deea0409da19dd238dd8"
server_key_iv: "52422f763d8cfd9b3324963dd9666d1e"
server_key_version: "encryptionkey@0"
I20260923 18:10:17.270792 7192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "bd2bf8cd10044e5a9d96771205299027"
format_stamp: "Formatted at 2026-09-23 18:10:17 on dist-test-slave-rvjj"
server_key: "7280bc95efa1deea0409da19dd238dd8"
server_key_iv: "52422f763d8cfd9b3324963dd9666d1e"
server_key_version: "encryptionkey@0"
I20260923 18:10:17.275481 7192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:10:17.278746 7207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:17.280616 7192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:17.280792 7192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "bd2bf8cd10044e5a9d96771205299027"
format_stamp: "Formatted at 2026-09-23 18:10:17 on dist-test-slave-rvjj"
server_key: "7280bc95efa1deea0409da19dd238dd8"
server_key_iv: "52422f763d8cfd9b3324963dd9666d1e"
server_key_version: "encryptionkey@0"
I20260923 18:10:17.280923 7192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:17.309888 7192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:17.313308 7192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:17.313503 7192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:17.321208 7192 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45501
I20260923 18:10:17.321396 7259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45501 every 8 connection(s)
I20260923 18:10:17.322304 7192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:17.324954 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7192
I20260923 18:10:17.325099 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:17.325477 18611 external_mini_cluster.cc:1428] Setting key 58aa96bfc58bf4c02e23f033f709a7f2
I20260923 18:10:17.326884 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:17.336548 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027: Bootstrap starting.
I20260923 18:10:17.338713 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:17.327428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:60141 (local address 127.0.0.1:45501)
0923 18:10:17.328035 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:17.328062 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:17.328122 (+ 60us) server_negotiation.cc:407] Connection header received
0923 18:10:17.329189 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:17.329218 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:17.330020 (+ 802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:17.330556 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:17.331223 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:17.332896 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:17.334007 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:17.334827 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:17.335045 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:17.336084 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:17.336091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:17.336114 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:17.336754 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:17.336845 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:17.336880 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:17.337068 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:17.337355 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 18:10:17.337537 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":163,"threads_started":1}
I20260923 18:10:17.340231 7260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:17.341434 7260 log.cc:824] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:17.343639 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:17.344659 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027: No bootstrap required, opened a new log
I20260923 18:10:17.349536 7260 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260923 18:10:17.350026 7260 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:17.350095 7260 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2bf8cd10044e5a9d96771205299027, State: Initialized, Role: FOLLOWER
I20260923 18:10:17.350881 7260 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260923 18:10:17.351195 7260 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:17.351347 7260 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:17.351588 7260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:17.351804 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:17.346071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:33487 (local address 127.0.0.1:45501)
0923 18:10:17.346237 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:17.346244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:17.346271 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:10:17.346423 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:17.346426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:17.346493 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:17.346622 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:17.346938 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:17.348325 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:17.349351 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:17.349714 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:17.349852 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:17.351027 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:17.351032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:17.351034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:17.351053 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:17.351097 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:17.351114 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:17.351247 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:17.351539 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 18:10:17.351603 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:10:17.353252 7260 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260923 18:10:17.353776 7260 leader_election.cc:302] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2bf8cd10044e5a9d96771205299027; no voters:
I20260923 18:10:17.354233 7260 leader_election.cc:288] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:17.354450 7270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:17.354763 7270 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [term 1 LEADER]: Becoming Leader. State: Replica: bd2bf8cd10044e5a9d96771205299027, State: Running, Role: LEADER
I20260923 18:10:17.355201 7270 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } }
I20260923 18:10:17.356166 7260 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:17.357128 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2bf8cd10044e5a9d96771205299027. Latest consensus state: current_term: 1 leader_uuid: "bd2bf8cd10044e5a9d96771205299027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260923 18:10:17.357395 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:17.357197 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2bf8cd10044e5a9d96771205299027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bf8cd10044e5a9d96771205299027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45501 } } }
I20260923 18:10:17.357667 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:17.357936 7279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:17.361629 7279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:17.369923 7279 catalog_manager.cc:1347] Generated new cluster ID: 254cc5e9966b4770b20513f76cb651fc
I20260923 18:10:17.370139 7279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:17.432344 7279 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:17.433691 7279 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:17.443104 7279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2bf8cd10044e5a9d96771205299027: Generated new TSK 0
I20260923 18:10:17.444492 7279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:17.456537 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7192
W20260923 18:10:17.464848 7177 connection.cc:535] client connection to 127.0.0.1:45501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45501): Connection reset by peer (error 104)
2026-09-23T18:10:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:17 dist-test-slave-rvjj krb5kdc[7291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:17 dist-test-slave-rvjj krb5kdc[7291](info): set up 2 sockets
Sep 23 18:10:17 dist-test-slave-rvjj krb5kdc[7291](info): commencing operation
krb5kdc: starting...
W20260923 18:10:18.437170 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:18 dist-test-slave-rvjj krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:18.584244 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:18.730554 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:18.731003 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:18.731072 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:18.735850 7304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:18.735970 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:18.735998 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:18.736021 7304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:18.736040 7304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:18.743809 7304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:18.745447 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:18.746948 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:18.759656 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:18.759737 7304 server_base.cc:1034] running on GCE node
W20260923 18:10:18.759693 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:18.759650 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:18.760903 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:18.762512 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:18.763866 7304 hybrid_clock.cc:648] HybridClock initialized: now 1790187018763748 us; error 127 us; skew 500 ppm
Sep 23 18:10:18 dist-test-slave-rvjj krb5kdc[7291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:18.767611 7304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:18.768918 7304 webserver.cc:460] Webserver started at http://127.0.0.1:36013/ using document root <none> and password file <none>
I20260923 18:10:18.769654 7304 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:18.769735 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:18.769979 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:18.772806 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "1dbc9e4771dd4b08b97109aa259f6b47"
format_stamp: "Formatted at 2026-09-23 18:10:18 on dist-test-slave-rvjj"
server_key: "424a3ffe6cc0ac81eca9103c9bb33bbc"
server_key_iv: "13a05c45e47c2d3ed52eedcd98017755"
server_key_version: "encryptionkey@0"
I20260923 18:10:18.773537 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "1dbc9e4771dd4b08b97109aa259f6b47"
format_stamp: "Formatted at 2026-09-23 18:10:18 on dist-test-slave-rvjj"
server_key: "424a3ffe6cc0ac81eca9103c9bb33bbc"
server_key_iv: "13a05c45e47c2d3ed52eedcd98017755"
server_key_version: "encryptionkey@0"
I20260923 18:10:18.778867 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 18:10:18.782538 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:18.784586 7304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:18.784785 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "1dbc9e4771dd4b08b97109aa259f6b47"
format_stamp: "Formatted at 2026-09-23 18:10:18 on dist-test-slave-rvjj"
server_key: "424a3ffe6cc0ac81eca9103c9bb33bbc"
server_key_iv: "13a05c45e47c2d3ed52eedcd98017755"
server_key_version: "encryptionkey@0"
I20260923 18:10:18.784935 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:18.791163 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:18.795969 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:18.796207 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:18.806131 7304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44547
I20260923 18:10:18.806160 7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44547 every 8 connection(s)
I20260923 18:10:18.807571 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:18.811013 7372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:18.813490 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7304
I20260923 18:10:18.813673 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:18.814005 18611 external_mini_cluster.cc:1428] Setting key 686015d446ea86abc6833a16b1991196
I20260923 18:10:18.818135 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47: Bootstrap starting.
I20260923 18:10:18.821081 7372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:18.822119 7372 log.cc:824] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:18.822952 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:18.816147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:44061 (local address 127.0.0.1:44547)
0923 18:10:18.816895 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:18.816916 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:18.816972 (+ 56us) server_negotiation.cc:407] Connection header received
0923 18:10:18.818124 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:18.818155 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:18.818669 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:18.819180 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:18.820234 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:18.820240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:18.820256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:18.820270 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:18.820342 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:18.820368 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:18.820497 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:18.820616 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 18:10:18.820854 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":197,"threads_started":1}
I20260923 18:10:18.825070 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47: No bootstrap required, opened a new log
I20260923 18:10:18.827870 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:18.829638 7372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260923 18:10:18.829932 7372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:18.829979 7372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbc9e4771dd4b08b97109aa259f6b47, State: Initialized, Role: FOLLOWER
I20260923 18:10:18.830540 7372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260923 18:10:18.830736 7372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:18.830818 7372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:18.830930 7372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:18.832139 7372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260923 18:10:18.832568 7372 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbc9e4771dd4b08b97109aa259f6b47; no voters:
I20260923 18:10:18.832945 7372 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:18.833097 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:18.830549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:53091 (local address 127.0.0.1:44547)
0923 18:10:18.830693 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:18.830699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:18.831131 (+ 432us) server_negotiation.cc:407] Connection header received
0923 18:10:18.831366 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:18.831369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:18.831422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:18.831500 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:18.832506 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:18.832511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:18.832514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:18.832527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:18.832566 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:18.832580 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:18.832689 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:18.832879 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 18:10:18.832960 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:10:18.833205 7382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:18.833683 7382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbc9e4771dd4b08b97109aa259f6b47, State: Running, Role: LEADER
I20260923 18:10:18.834219 7382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260923 18:10:18.834599 7372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:18.836347 7383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } } }
I20260923 18:10:18.836397 7384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbc9e4771dd4b08b97109aa259f6b47. Latest consensus state: current_term: 1 leader_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbc9e4771dd4b08b97109aa259f6b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } } }
I20260923 18:10:18.836644 7384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:18.836979 7383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:18.837255 7396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:18.840998 7396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:18.849134 7396 catalog_manager.cc:1347] Generated new cluster ID: e42584f6349a422eb0a6e0c76f9a5d9d
I20260923 18:10:18.849323 7396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:18.867202 7396 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:18.867631 7396 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:18.879065 7396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbc9e4771dd4b08b97109aa259f6b47: Generated new TSK 0
I20260923 18:10:18.879971 7396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:18.884655 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7304
2026-09-23T18:10:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:18 dist-test-slave-rvjj krb5kdc[7403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:18 dist-test-slave-rvjj krb5kdc[7403](info): set up 2 sockets
Sep 23 18:10:18 dist-test-slave-rvjj krb5kdc[7403](info): commencing operation
krb5kdc: starting...
W20260923 18:10:19.880656 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:19 dist-test-slave-rvjj krb5kdc[7403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:20.029438 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:20.172461 7416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:20.172792 7416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:20.172839 7416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:20.177239 7416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:20.177338 7416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:20.177359 7416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:20.177376 7416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:20.177390 7416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:20.183197 7416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:20.184540 7416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:20.185688 7416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:20.197198 7424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:20.197327 7422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:20.197342 7421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:20.197465 7416 server_base.cc:1034] running on GCE node
I20260923 18:10:20.198345 7416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:20.199690 7416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:20.201087 7416 hybrid_clock.cc:648] HybridClock initialized: now 1790187020201058 us; error 65 us; skew 500 ppm
Sep 23 18:10:20 dist-test-slave-rvjj krb5kdc[7403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:20.204895 7416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:20.206432 7416 webserver.cc:460] Webserver started at http://127.0.0.1:33709/ using document root <none> and password file <none>
I20260923 18:10:20.207224 7416 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:20.207357 7416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:20.207634 7416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:20.210505 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "3ff71e93e75d4076817ed99bbb694c6c"
format_stamp: "Formatted at 2026-09-23 18:10:20 on dist-test-slave-rvjj"
server_key: "5cf512eaefc4fa2bf04e71ecfcb4c3ed"
server_key_iv: "bfc3d05ea44bbfeaf1c95118babde229"
server_key_version: "encryptionkey@0"
I20260923 18:10:20.211402 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "3ff71e93e75d4076817ed99bbb694c6c"
format_stamp: "Formatted at 2026-09-23 18:10:20 on dist-test-slave-rvjj"
server_key: "5cf512eaefc4fa2bf04e71ecfcb4c3ed"
server_key_iv: "bfc3d05ea44bbfeaf1c95118babde229"
server_key_version: "encryptionkey@0"
I20260923 18:10:20.217118 7416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:20.221329 7431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:20.223536 7416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:20.223767 7416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "3ff71e93e75d4076817ed99bbb694c6c"
format_stamp: "Formatted at 2026-09-23 18:10:20 on dist-test-slave-rvjj"
server_key: "5cf512eaefc4fa2bf04e71ecfcb4c3ed"
server_key_iv: "bfc3d05ea44bbfeaf1c95118babde229"
server_key_version: "encryptionkey@0"
I20260923 18:10:20.223966 7416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:20.263715 7416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:20.268316 7416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:20.268574 7416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:20.279563 7416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41863
I20260923 18:10:20.279600 7483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41863 every 8 connection(s)
I20260923 18:10:20.281076 7416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:20.284610 7484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:20.290313 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7416
I20260923 18:10:20.290470 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:20.290639 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: Bootstrap starting.
I20260923 18:10:20.290815 18611 external_mini_cluster.cc:1428] Setting key 76df38c0c5eed001da645bc6d69ee9c7
I20260923 18:10:20.293497 7484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:20.294277 7484 log.cc:824] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:20.297266 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: No bootstrap required, opened a new log
I20260923 18:10:20.301851 7484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260923 18:10:20.302237 7484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:20.302290 7484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff71e93e75d4076817ed99bbb694c6c, State: Initialized, Role: FOLLOWER
I20260923 18:10:20.302886 7484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260923 18:10:20.303084 7484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:20.303170 7484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:20.303313 7484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 0 FOLLOWER]: Advancing to term 1
Sep 23 18:10:20 dist-test-slave-rvjj krb5kdc[7403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:10:20.304603 7484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260923 18:10:20.304991 7484 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ff71e93e75d4076817ed99bbb694c6c; no voters:
I20260923 18:10:20.305364 7484 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:20.305579 7489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:20.305999 7489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [term 1 LEADER]: Becoming Leader. State: Replica: 3ff71e93e75d4076817ed99bbb694c6c, State: Running, Role: LEADER
I20260923 18:10:20.306525 7489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260923 18:10:20.306919 7484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:20.308379 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ff71e93e75d4076817ed99bbb694c6c. Latest consensus state: current_term: 1 leader_uuid: "3ff71e93e75d4076817ed99bbb694c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } } }
I20260923 18:10:20.308552 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:20.308799 7490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ff71e93e75d4076817ed99bbb694c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff71e93e75d4076817ed99bbb694c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } } }
I20260923 18:10:20.308939 7490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:20.312297 7504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:20.312379 7504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:20.312431 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:20.292712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:60885 (local address 127.0.0.1:41863)
0923 18:10:20.293567 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:20.293592 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:20.293660 (+ 68us) server_negotiation.cc:407] Connection header received
0923 18:10:20.294765 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:20.294803 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:20.295506 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:20.295955 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:20.297081 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:20.298422 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:20.299857 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:20.300638 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:20.300853 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:20.304741 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:20.304752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:20.304787 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:20.304851 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:20.307501 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:20.308474 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:20.308481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:20.308488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:20.308680 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:20.309166 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:20.309170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:20.309172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:20.309856 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:20.310138 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:20.310504 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 18:10:20.310838 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":627,"thread_start_us":162,"threads_started":1}
I20260923 18:10:20.312451 7501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:20.313346 7501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:20.316790 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:20.319545 7501 catalog_manager.cc:1347] Generated new cluster ID: aa02878ff5294ca4b33cab0ab1c0d125
I20260923 18:10:20.319645 7501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:20.331400 7501 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:20.333266 7501 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:20.335047 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:20.319957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:49745 (local address 127.0.0.1:41863)
0923 18:10:20.320074 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:20.320079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:20.323958 (+ 3879us) server_negotiation.cc:407] Connection header received
0923 18:10:20.324133 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:20.324137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:20.324226 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:20.324506 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:20.325667 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:20.326660 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:20.328145 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:20.328510 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:20.328642 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:20.331095 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:20.331099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:20.331102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:20.331145 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:20.332759 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:20.333394 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:20.333397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:20.333400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:20.333488 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:20.333822 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:20.333825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:20.333827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:20.334049 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:20.334190 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:20.334703 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 18:10:20.334874 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 18:10:20.350059 7501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ff71e93e75d4076817ed99bbb694c6c: Generated new TSK 0
I20260923 18:10:20.350976 7501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:20.359413 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7416
W20260923 18:10:20.365207 7401 connection.cc:535] client connection to 127.0.0.1:41863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41863): Connection reset by peer (error 104)
2026-09-23T18:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:20 dist-test-slave-rvjj krb5kdc[7515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:20 dist-test-slave-rvjj krb5kdc[7515](info): set up 2 sockets
Sep 23 18:10:20 dist-test-slave-rvjj krb5kdc[7515](info): commencing operation
krb5kdc: starting...
W20260923 18:10:21.364836 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:21 dist-test-slave-rvjj krb5kdc[7515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:21.516119 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:21.633644 7528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:21.633967 7528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:21.634027 7528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:21.638346 7528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:21.638458 7528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:21.638486 7528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:21.638507 7528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:21.638527 7528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:21.644366 7528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:21.645568 7528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:21.646664 7528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:21.657502 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:21.657601 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:21.657492 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:21.658581 7528 server_base.cc:1034] running on GCE node
I20260923 18:10:21.659442 7528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:21.661360 7528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:21.662647 7528 hybrid_clock.cc:648] HybridClock initialized: now 1790187021662603 us; error 64 us; skew 500 ppm
Sep 23 18:10:21 dist-test-slave-rvjj krb5kdc[7515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:21.666780 7528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:21.668362 7528 webserver.cc:460] Webserver started at http://127.0.0.1:39665/ using document root <none> and password file <none>
I20260923 18:10:21.669169 7528 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:21.669256 7528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:21.669507 7528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:21.672452 7528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68"
format_stamp: "Formatted at 2026-09-23 18:10:21 on dist-test-slave-rvjj"
server_key: "3971aa86eed29fecf7a4ec8c2e46881b"
server_key_iv: "44af8e3b3698d30e99d42d305e8a8d10"
server_key_version: "encryptionkey@0"
I20260923 18:10:21.673237 7528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68"
format_stamp: "Formatted at 2026-09-23 18:10:21 on dist-test-slave-rvjj"
server_key: "3971aa86eed29fecf7a4ec8c2e46881b"
server_key_iv: "44af8e3b3698d30e99d42d305e8a8d10"
server_key_version: "encryptionkey@0"
I20260923 18:10:21.678869 7528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:10:21.682885 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:21.684900 7528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:21.685122 7528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68"
format_stamp: "Formatted at 2026-09-23 18:10:21 on dist-test-slave-rvjj"
server_key: "3971aa86eed29fecf7a4ec8c2e46881b"
server_key_iv: "44af8e3b3698d30e99d42d305e8a8d10"
server_key_version: "encryptionkey@0"
I20260923 18:10:21.685292 7528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:21.702226 7528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:21.706475 7528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:21.706740 7528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:21.716562 7528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33699
I20260923 18:10:21.716624 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33699 every 8 connection(s)
I20260923 18:10:21.717897 7528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:21.721191 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:21.725623 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7528
I20260923 18:10:21.725777 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:21.726195 18611 external_mini_cluster.cc:1428] Setting key 135b80acc4f8b5c6dd8ec6a6046ca231
I20260923 18:10:21.728295 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: Bootstrap starting.
I20260923 18:10:21.730983 7596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:21.732079 7596 log.cc:824] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:21.735232 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: No bootstrap required, opened a new log
I20260923 18:10:21.739548 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:21.728576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:59645 (local address 127.0.0.1:33699)
0923 18:10:21.729197 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:21.729219 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:21.729270 (+ 51us) server_negotiation.cc:407] Connection header received
0923 18:10:21.730194 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:21.730219 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:21.730747 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:21.731193 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:21.731747 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:21.733091 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:21.734623 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:21.735410 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:21.735613 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:21.736737 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:21.736744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:21.736765 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:21.737367 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:21.737457 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:21.737532 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:21.737677 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:21.738032 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 18:10:21.738332 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":188,"threads_started":1}
I20260923 18:10:21.739622 7596 raft_consensus.cc:348] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260923 18:10:21.739970 7596 raft_consensus.cc:374] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:21.740032 7596 raft_consensus.cc:729] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeab6fd25de43e4a9c8ba0d84bb5a68, State: Initialized, Role: FOLLOWER
I20260923 18:10:21.740522 7596 consensus_queue.cc:260] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260923 18:10:21.740672 7596 raft_consensus.cc:388] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:21.740743 7596 raft_consensus.cc:482] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:21.740849 7596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:21.742213 7596 raft_consensus.cc:504] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260923 18:10:21.742651 7596 leader_election.cc:302] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baeab6fd25de43e4a9c8ba0d84bb5a68; no voters:
I20260923 18:10:21.743080 7596 leader_election.cc:288] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:21.743431 7601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:21.743913 7601 raft_consensus.cc:686] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [term 1 LEADER]: Becoming Leader. State: Replica: baeab6fd25de43e4a9c8ba0d84bb5a68, State: Running, Role: LEADER
I20260923 18:10:21.743963 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:21.744649 7596 sys_catalog.cc:564] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:21.744614 7601 consensus_queue.cc:237] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260923 18:10:21.746888 7602 sys_catalog.cc:455] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } } }
I20260923 18:10:21.748212 7602 sys_catalog.cc:458] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:21.747169 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baeab6fd25de43e4a9c8ba0d84bb5a68. Latest consensus state: current_term: 1 leader_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baeab6fd25de43e4a9c8ba0d84bb5a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } } }
I20260923 18:10:21.748486 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:21.749996 7620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:21.750085 7620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:21.751228 7621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:21.752108 7621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:21.753710 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:21.747513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:54551 (local address 127.0.0.1:33699)
0923 18:10:21.747659 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:21.747665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:21.747689 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:10:21.747743 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:21.747746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:21.747815 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:21.747935 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:21.748514 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:21.749342 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:21.750593 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:21.750878 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:21.750988 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:21.752707 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:21.752713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:21.752716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:21.752736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:21.752783 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:21.752800 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:21.752893 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:21.753468 (+ 575us) server_negotiation.cc:299] Negotiation successful
0923 18:10:21.753540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 18:10:21.759856 7621 catalog_manager.cc:1347] Generated new cluster ID: 509a05c74dca43e48b3a3ce69ac2541c
I20260923 18:10:21.760062 7621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:21.786512 7621 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:21.787717 7621 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:21.801327 7621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baeab6fd25de43e4a9c8ba0d84bb5a68: Generated new TSK 0
I20260923 18:10:21.802314 7621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:21.815896 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7528
W20260923 18:10:21.822646 7513 connection.cc:535] client connection to 127.0.0.1:33699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33699): Connection reset by peer (error 104)
2026-09-23T18:10:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:21 dist-test-slave-rvjj krb5kdc[7627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:21 dist-test-slave-rvjj krb5kdc[7627](info): set up 2 sockets
Sep 23 18:10:21 dist-test-slave-rvjj krb5kdc[7627](info): commencing operation
krb5kdc: starting...
W20260923 18:10:22.805145 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:22 dist-test-slave-rvjj krb5kdc[7627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:10:22.951464 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:23.078832 7640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:23.079156 7640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:23.079203 7640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:23.083546 7640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:23.083642 7640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:23.083662 7640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:23.083678 7640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:23.083693 7640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:23.089556 7640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:23.090708 7640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:23.091805 7640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:23.101859 7646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:23.101881 7645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:23.101874 7648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:23.102240 7640 server_base.cc:1034] running on GCE node
I20260923 18:10:23.103057 7640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:23.104423 7640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:23.105640 7640 hybrid_clock.cc:648] HybridClock initialized: now 1790187023105617 us; error 56 us; skew 500 ppm
Sep 23 18:10:23 dist-test-slave-rvjj krb5kdc[7627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:23.113888 7640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:10:23.115043 7640 webserver.cc:460] Webserver started at http://127.0.0.1:39763/ using document root <none> and password file <none>
I20260923 18:10:23.115712 7640 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:23.115788 7640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:23.115998 7640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:23.118259 7640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "f84855ebbd7f4cbfbe80371f20acc1f3"
format_stamp: "Formatted at 2026-09-23 18:10:23 on dist-test-slave-rvjj"
server_key: "718ec7e8ef94c1755e04ec6f64058ef3"
server_key_iv: "2f9088e7091f704b059389ade7fc8a32"
server_key_version: "encryptionkey@0"
I20260923 18:10:23.118845 7640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "f84855ebbd7f4cbfbe80371f20acc1f3"
format_stamp: "Formatted at 2026-09-23 18:10:23 on dist-test-slave-rvjj"
server_key: "718ec7e8ef94c1755e04ec6f64058ef3"
server_key_iv: "2f9088e7091f704b059389ade7fc8a32"
server_key_version: "encryptionkey@0"
I20260923 18:10:23.123165 7640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:10:23.126307 7655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:23.127930 7640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 18:10:23.128069 7640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "f84855ebbd7f4cbfbe80371f20acc1f3"
format_stamp: "Formatted at 2026-09-23 18:10:23 on dist-test-slave-rvjj"
server_key: "718ec7e8ef94c1755e04ec6f64058ef3"
server_key_iv: "2f9088e7091f704b059389ade7fc8a32"
server_key_version: "encryptionkey@0"
I20260923 18:10:23.128188 7640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:23.134091 7640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:23.138176 7640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:23.138377 7640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:23.146507 7640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36181
I20260923 18:10:23.146538 7707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36181 every 8 connection(s)
I20260923 18:10:23.147699 7640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:23.150513 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7640
I20260923 18:10:23.150662 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:23.151012 18611 external_mini_cluster.cc:1428] Setting key 5ba4edc2c5beeb5f742ec6454e2fa4d9
I20260923 18:10:23.151752 7708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:23.157294 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:23.152717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:51507 (local address 127.0.0.1:36181)
0923 18:10:23.153465 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:23.153489 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:23.153533 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:10:23.154314 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:23.154338 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:23.154649 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:23.154963 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:23.155846 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:23.155851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:23.155863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:23.155873 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:23.155925 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:23.155946 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:23.156064 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:23.156177 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 18:10:23.156465 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":164,"threads_started":1}
I20260923 18:10:23.159654 7708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: Bootstrap starting.
I20260923 18:10:23.160717 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:23.162433 7708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:23.163422 7708 log.cc:824] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:23.165480 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:23.163282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:59517 (local address 127.0.0.1:36181)
0923 18:10:23.163408 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:23.163413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:23.163604 (+ 191us) server_negotiation.cc:407] Connection header received
0923 18:10:23.163741 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:23.163744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:23.163793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:23.163887 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:23.164977 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:23.164981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:23.164983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:23.164994 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:23.165019 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:23.165029 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:23.165121 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:23.165225 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 18:10:23.165353 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 18:10:23.166046 7708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: No bootstrap required, opened a new log
I20260923 18:10:23.170014 7708 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260923 18:10:23.170321 7708 raft_consensus.cc:374] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:23.170375 7708 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84855ebbd7f4cbfbe80371f20acc1f3, State: Initialized, Role: FOLLOWER
I20260923 18:10:23.170845 7708 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260923 18:10:23.171021 7708 raft_consensus.cc:388] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:23.171094 7708 raft_consensus.cc:482] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:23.171204 7708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:23.172446 7708 raft_consensus.cc:504] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260923 18:10:23.172798 7708 leader_election.cc:302] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f84855ebbd7f4cbfbe80371f20acc1f3; no voters:
I20260923 18:10:23.173108 7708 leader_election.cc:288] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:23.173271 7718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:23.173512 7718 raft_consensus.cc:686] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [term 1 LEADER]: Becoming Leader. State: Replica: f84855ebbd7f4cbfbe80371f20acc1f3, State: Running, Role: LEADER
I20260923 18:10:23.173923 7718 consensus_queue.cc:237] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } }
I20260923 18:10:23.174412 7708 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:23.175591 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } } }
I20260923 18:10:23.175736 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:23.177655 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f84855ebbd7f4cbfbe80371f20acc1f3. Latest consensus state: current_term: 1 leader_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84855ebbd7f4cbfbe80371f20acc1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36181 } } }
I20260923 18:10:23.177780 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3 [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:23.179517 7733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:23.179612 7733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:23.179735 7727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:23.180516 7727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:23.187299 7727 catalog_manager.cc:1347] Generated new cluster ID: 15eed62b8b8b47ba93b8d814060d102d
I20260923 18:10:23.187459 7727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:23.202980 7727 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:23.203395 7727 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:23.216019 7727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f84855ebbd7f4cbfbe80371f20acc1f3: Generated new TSK 0
I20260923 18:10:23.216892 7727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:23.224092 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7640
2026-09-23T18:10:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:23 dist-test-slave-rvjj krb5kdc[7739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:23 dist-test-slave-rvjj krb5kdc[7739](info): set up 2 sockets
Sep 23 18:10:23 dist-test-slave-rvjj krb5kdc[7739](info): commencing operation
krb5kdc: starting...
W20260923 18:10:24.228678 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:24.372094 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:32983
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:32983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:24.533063 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:24.533964 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:24.534097 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:24.544826 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:24.544981 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:24.545011 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:24.545037 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:24.545265 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:24.555060 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:32983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:24.556824 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:24.558352 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:24.571453 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:24.571599 7752 server_base.cc:1034] running on GCE node
W20260923 18:10:24.571475 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:24.571602 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:24.573164 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:24.574608 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:24.578042 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790187024577936 us; error 221 us; skew 500 ppm
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187024, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:24.582657 7752 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:24.584275 7752 webserver.cc:460] Webserver started at http://127.18.44.254:40171/ using document root <none> and password file <none>
I20260923 18:10:24.585104 7752 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:24.585203 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:24.585467 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:24.588768 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "85bf747b086e467a9fd2d063b4629170"
format_stamp: "Formatted at 2026-09-23 18:10:24 on dist-test-slave-rvjj"
server_key: "e50576fc6f8c8a3f6d4e9ae680eb7d77"
server_key_iv: "c906783264e571d0687567b3cd410acf"
server_key_version: "encryptionkey@0"
I20260923 18:10:24.589620 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "85bf747b086e467a9fd2d063b4629170"
format_stamp: "Formatted at 2026-09-23 18:10:24 on dist-test-slave-rvjj"
server_key: "e50576fc6f8c8a3f6d4e9ae680eb7d77"
server_key_iv: "c906783264e571d0687567b3cd410acf"
server_key_version: "encryptionkey@0"
I20260923 18:10:24.595482 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:10:24.600023 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:24.602326 7752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:24.602574 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "85bf747b086e467a9fd2d063b4629170"
format_stamp: "Formatted at 2026-09-23 18:10:24 on dist-test-slave-rvjj"
server_key: "e50576fc6f8c8a3f6d4e9ae680eb7d77"
server_key_iv: "c906783264e571d0687567b3cd410acf"
server_key_version: "encryptionkey@0"
I20260923 18:10:24.602763 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:24.622666 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:24.628039 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:24.628331 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:24.639673 7752 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:32983
I20260923 18:10:24.639796 7819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:32983 every 8 connection(s)
I20260923 18:10:24.641214 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:24.643023 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7752
I20260923 18:10:24.643164 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:24.643519 18611 external_mini_cluster.cc:1428] Setting key cf2f5cd645a6a0154764b0ccaac1575d
I20260923 18:10:24.645193 7820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:24.652954 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: Bootstrap starting.
I20260923 18:10:24.655953 7820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: Neither blocks nor log segments found. Creating new log.
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:24.657058 7820 log.cc:824] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:24.660691 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: No bootstrap required, opened a new log
I20260923 18:10:24.667554 7820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } }
I20260923 18:10:24.667896 7820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:24.667946 7820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bf747b086e467a9fd2d063b4629170, State: Initialized, Role: FOLLOWER
I20260923 18:10:24.668479 7820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } }
I20260923 18:10:24.668637 7820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:24.668706 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:24.668828 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:24.668993 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:24.645862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51335 (local address 127.18.44.254:32983)
0923 18:10:24.646534 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:24.646556 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:24.646611 (+ 55us) server_negotiation.cc:407] Connection header received
0923 18:10:24.647601 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:24.647629 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:24.648133 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:24.648588 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:24.649612 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:24.651127 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:24.652481 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:24.653257 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:24.653498 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:24.657544 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:24.657553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:24.657581 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:24.657659 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:24.662696 (+ 5037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:24.664112 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:24.664120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:24.664127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:24.664598 (+ 471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:24.665279 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:24.665285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:24.665286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:24.666107 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:24.666415 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:24.667101 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 18:10:24.667583 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":225,"threads_started":1}
I20260923 18:10:24.670389 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } }
I20260923 18:10:24.670918 7820 leader_election.cc:302] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bf747b086e467a9fd2d063b4629170; no voters:
I20260923 18:10:24.671702 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:24.671799 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:24.672142 7825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [term 1 LEADER]: Becoming Leader. State: Replica: 85bf747b086e467a9fd2d063b4629170, State: Running, Role: LEADER
I20260923 18:10:24.672569 7825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } }
I20260923 18:10:24.674044 7820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:24.674958 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bf747b086e467a9fd2d063b4629170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } } }
I20260923 18:10:24.676003 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bf747b086e467a9fd2d063b4629170. Latest consensus state: current_term: 1 leader_uuid: "85bf747b086e467a9fd2d063b4629170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bf747b086e467a9fd2d063b4629170" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 32983 } } }
I20260923 18:10:24.677914 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:24.675819 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:24.680423 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 18:10:24.680516 7839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:24.680585 7839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:24.680670 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:24.681694 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:24.692499 7842 catalog_manager.cc:1347] Generated new cluster ID: c75ef6b4486b451cb223acb77a272e80
I20260923 18:10:24.692667 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:24.697485 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:24.684553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46587 (local address 127.18.44.254:32983)
0923 18:10:24.685153 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:24.685159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:24.685472 (+ 313us) server_negotiation.cc:407] Connection header received
0923 18:10:24.685676 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:24.685680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:24.685747 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:24.685882 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:24.687489 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:24.688608 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:24.689950 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:24.690214 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:24.690329 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:24.692491 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:24.692495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:24.692498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:24.692540 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:24.694745 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:24.695672 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:24.695677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:24.695679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:24.695790 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:24.696200 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:24.696204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:24.696206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:24.696428 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:24.696574 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:24.697168 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 18:10:24.697301 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499}
I20260923 18:10:24.705415 7842 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:24.707353 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:24.715904 7842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bf747b086e467a9fd2d063b4629170: Generated new TSK 0
I20260923 18:10:24.716797 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:24.722146 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7752
W20260923 18:10:24.728158 7737 connection.cc:535] client connection to 127.18.44.254:32983 recv error: Network error: failed to read from TLS socket (remote: 127.18.44.254:32983): Connection reset by peer (error 104)
2026-09-23T18:10:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7851](info): set up 2 sockets
Sep 23 18:10:24 dist-test-slave-rvjj krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260923 18:10:25.768514 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.010s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:25 dist-test-slave-rvjj krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:25.941009 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46351
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:26.070142 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:26.070459 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:26.070502 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:26.074777 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:26.074868 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:26.074888 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:26.074903 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:26.074918 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:26.080592 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:26.081811 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:26.082895 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:26.093127 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:26.093127 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:26.093405 7864 server_base.cc:1034] running on GCE node
W20260923 18:10:26.093217 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:26.094282 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:26.095997 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:26.097263 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790187026097229 us; error 58 us; skew 500 ppm
Sep 23 18:10:26 dist-test-slave-rvjj krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:26.101771 7864 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:26.103346 7864 webserver.cc:460] Webserver started at http://127.18.44.254:39193/ using document root <none> and password file <none>
I20260923 18:10:26.104149 7864 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:26.104243 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:26.104511 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:26.107755 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "0af58e62bb564ff8a08c6e840e53b94e"
format_stamp: "Formatted at 2026-09-23 18:10:26 on dist-test-slave-rvjj"
server_key: "a601aaf0460077254d8dfa3ccc7eb844"
server_key_iv: "225efb3d027103abdd4fbda44a3014d8"
server_key_version: "encryptionkey@0"
I20260923 18:10:26.108595 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "0af58e62bb564ff8a08c6e840e53b94e"
format_stamp: "Formatted at 2026-09-23 18:10:26 on dist-test-slave-rvjj"
server_key: "a601aaf0460077254d8dfa3ccc7eb844"
server_key_iv: "225efb3d027103abdd4fbda44a3014d8"
server_key_version: "encryptionkey@0"
I20260923 18:10:26.114675 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 18:10:26.118949 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:26.121220 7864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:26.121582 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "0af58e62bb564ff8a08c6e840e53b94e"
format_stamp: "Formatted at 2026-09-23 18:10:26 on dist-test-slave-rvjj"
server_key: "a601aaf0460077254d8dfa3ccc7eb844"
server_key_iv: "225efb3d027103abdd4fbda44a3014d8"
server_key_version: "encryptionkey@0"
I20260923 18:10:26.121824 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:26.169274 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:26.173699 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:26.173972 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:26.183826 7864 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:46351
I20260923 18:10:26.183902 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:46351 every 8 connection(s)
I20260923 18:10:26.184978 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:26.189191 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:26.191634 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7864
I20260923 18:10:26.191799 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:26.192116 18611 external_mini_cluster.cc:1428] Setting key 8c2b80da6c2a5d0f67a7d016e654926e
I20260923 18:10:26.198333 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e: Bootstrap starting.
I20260923 18:10:26.201545 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:26.202551 7932 log.cc:824] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:26.204006 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:26.194287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41815 (local address 127.18.44.254:46351)
0923 18:10:26.194840 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:26.194858 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:26.194898 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:10:26.195821 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:26.195845 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:26.196255 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:26.196598 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:26.197292 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:26.198480 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:26.199684 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:26.200245 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:26.200469 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:26.201737 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:26.201745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:26.201774 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:26.202282 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:26.202357 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:26.202384 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:26.202518 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:26.202735 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 18:10:26.202928 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":160,"threads_started":1}
I20260923 18:10:26.205621 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e: No bootstrap required, opened a new log
I20260923 18:10:26.207962 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:26.210417 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } }
I20260923 18:10:26.210839 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:26.210902 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af58e62bb564ff8a08c6e840e53b94e, State: Initialized, Role: FOLLOWER
I20260923 18:10:26.211844 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } }
I20260923 18:10:26.212327 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:26.212428 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:26.212560 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:26.213991 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } }
I20260923 18:10:26.214573 7932 leader_election.cc:302] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af58e62bb564ff8a08c6e840e53b94e; no voters:
I20260923 18:10:26.215072 7932 leader_election.cc:288] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:26.215574 7942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:26.216002 7942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [term 1 LEADER]: Becoming Leader. State: Replica: 0af58e62bb564ff8a08c6e840e53b94e, State: Running, Role: LEADER
I20260923 18:10:26.216270 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:26.210231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40083 (local address 127.18.44.254:46351)
0923 18:10:26.210384 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:26.210389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:26.210566 (+ 177us) server_negotiation.cc:407] Connection header received
0923 18:10:26.210692 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:26.210696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:26.210771 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:26.210909 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:26.211791 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:26.212863 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:26.213947 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:26.214315 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:26.214461 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:26.215646 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:26.215652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:26.215655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:26.215674 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:26.215720 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:26.215739 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:26.215826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:26.216076 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 18:10:26.216131 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:10:26.216533 7942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } }
I20260923 18:10:26.216970 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:26.218528 7944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af58e62bb564ff8a08c6e840e53b94e. Latest consensus state: current_term: 1 leader_uuid: "0af58e62bb564ff8a08c6e840e53b94e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } } }
I20260923 18:10:26.218686 7944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:26.219049 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af58e62bb564ff8a08c6e840e53b94e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af58e62bb564ff8a08c6e840e53b94e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46351 } } }
I20260923 18:10:26.219142 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:26.219529 7953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:26.223448 7953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:26.231979 7953 catalog_manager.cc:1347] Generated new cluster ID: 7669d12e310f4a5c833fe828dbf375fe
I20260923 18:10:26.232110 7953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:26.254410 7953 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:26.256038 7953 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:26.268378 7953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af58e62bb564ff8a08c6e840e53b94e: Generated new TSK 0
I20260923 18:10:26.269425 7953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:26.276362 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7864
W20260923 18:10:26.281937 7849 connection.cc:535] client connection to 127.18.44.254:46351 recv error: Network error: failed to read from TLS socket (remote: 127.18.44.254:46351): Connection reset by peer (error 104)
2026-09-23T18:10:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:26 dist-test-slave-rvjj krb5kdc[7963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:26 dist-test-slave-rvjj krb5kdc[7963](info): set up 2 sockets
Sep 23 18:10:26 dist-test-slave-rvjj krb5kdc[7963](info): commencing operation
krb5kdc: starting...
W20260923 18:10:27.308965 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:27 dist-test-slave-rvjj krb5kdc[7963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:27.448928 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46405
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:27.573717 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:27.574026 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:27.574070 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:27.578291 7976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:27.578388 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:27.578414 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:27.578429 7976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:27.578444 7976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:27.584313 7976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:27.585513 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:27.586597 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:27.595847 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:27.595906 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:27.596081 7976 server_base.cc:1034] running on GCE node
W20260923 18:10:27.596194 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:27.596891 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:27.598135 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:27.599349 7976 hybrid_clock.cc:648] HybridClock initialized: now 1790187027599330 us; error 37 us; skew 500 ppm
Sep 23 18:10:27 dist-test-slave-rvjj krb5kdc[7963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187027, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:27.602478 7976 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:27.603675 7976 webserver.cc:460] Webserver started at http://127.18.44.254:41561/ using document root <none> and password file <none>
I20260923 18:10:27.604269 7976 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:27.604343 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:27.604528 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:27.606705 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "7857a21e5e4c47dc8be39d879696de35"
format_stamp: "Formatted at 2026-09-23 18:10:27 on dist-test-slave-rvjj"
server_key: "5f3706a240dae111dc79d641af3e611c"
server_key_iv: "f5eaecc517a29ef6e740718ef97e7b23"
server_key_version: "encryptionkey@0"
I20260923 18:10:27.607318 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "7857a21e5e4c47dc8be39d879696de35"
format_stamp: "Formatted at 2026-09-23 18:10:27 on dist-test-slave-rvjj"
server_key: "5f3706a240dae111dc79d641af3e611c"
server_key_iv: "f5eaecc517a29ef6e740718ef97e7b23"
server_key_version: "encryptionkey@0"
I20260923 18:10:27.611621 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:10:27.614651 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:27.616413 7976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:27.616586 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "7857a21e5e4c47dc8be39d879696de35"
format_stamp: "Formatted at 2026-09-23 18:10:27 on dist-test-slave-rvjj"
server_key: "5f3706a240dae111dc79d641af3e611c"
server_key_iv: "f5eaecc517a29ef6e740718ef97e7b23"
server_key_version: "encryptionkey@0"
I20260923 18:10:27.616734 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:27.621897 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:27.625674 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:27.625965 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:27.634694 7976 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:46405
I20260923 18:10:27.634769 8043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:46405 every 8 connection(s)
I20260923 18:10:27.635789 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:27.636878 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 7976
I20260923 18:10:27.637023 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:27.637321 18611 external_mini_cluster.cc:1428] Setting key 751d2c886af0cb3bf653fc6b85144b36
I20260923 18:10:27.639562 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:27.644011 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:27.638908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51367 (local address 127.18.44.254:46405)
0923 18:10:27.639521 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:27.639543 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:27.639621 (+ 78us) server_negotiation.cc:407] Connection header received
0923 18:10:27.640630 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:27.640655 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:27.641059 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:27.641507 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:27.642342 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:27.642349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:27.642365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:27.642378 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:27.642450 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:27.642477 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:27.642661 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:27.642737 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 18:10:27.642946 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":160,"threads_started":1}
I20260923 18:10:27.647441 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35: Bootstrap starting.
I20260923 18:10:27.647863 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:27.649998 8044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:27.650992 8044 log.cc:824] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:27.651827 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:27.650151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33227 (local address 127.18.44.254:46405)
0923 18:10:27.650297 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:27.650303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:27.650404 (+ 101us) server_negotiation.cc:407] Connection header received
0923 18:10:27.650522 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:27.650526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:27.650569 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:27.650664 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:27.651401 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:27.651405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:27.651407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:27.651421 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:27.651451 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:27.651465 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:27.651543 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:27.651662 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 18:10:27.651699 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:10:27.653426 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35: No bootstrap required, opened a new log
I20260923 18:10:27.657384 8044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } }
I20260923 18:10:27.657673 8044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:27.657727 8044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7857a21e5e4c47dc8be39d879696de35, State: Initialized, Role: FOLLOWER
I20260923 18:10:27.658154 8044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } }
I20260923 18:10:27.658283 8044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:27.658324 8044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:27.658403 8044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:27.659677 8044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } }
I20260923 18:10:27.660048 8044 leader_election.cc:302] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7857a21e5e4c47dc8be39d879696de35; no voters:
I20260923 18:10:27.660365 8044 leader_election.cc:288] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:27.660497 8054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:27.660730 8054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [term 1 LEADER]: Becoming Leader. State: Replica: 7857a21e5e4c47dc8be39d879696de35, State: Running, Role: LEADER
I20260923 18:10:27.661114 8054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } }
I20260923 18:10:27.661765 8044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:27.662915 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7857a21e5e4c47dc8be39d879696de35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } } }
I20260923 18:10:27.662925 8056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7857a21e5e4c47dc8be39d879696de35. Latest consensus state: current_term: 1 leader_uuid: "7857a21e5e4c47dc8be39d879696de35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7857a21e5e4c47dc8be39d879696de35" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46405 } } }
I20260923 18:10:27.663117 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:27.663183 8056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:27.663704 8063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:27.667410 8063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:27.674081 8063 catalog_manager.cc:1347] Generated new cluster ID: 5df63dee5d564a2096f10db100e4264c
I20260923 18:10:27.674191 8063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:27.693203 8063 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:27.693820 8063 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:27.704531 8063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7857a21e5e4c47dc8be39d879696de35: Generated new TSK 0
I20260923 18:10:27.705392 8063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:27.718855 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 7976
2026-09-23T18:10:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:27 dist-test-slave-rvjj krb5kdc[8075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:27 dist-test-slave-rvjj krb5kdc[8075](info): set up 2 sockets
Sep 23 18:10:27 dist-test-slave-rvjj krb5kdc[8075](info): commencing operation
krb5kdc: starting...
W20260923 18:10:28.721048 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:28 dist-test-slave-rvjj krb5kdc[8075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:28.872550 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:38565
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:29.017629 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:29.018085 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:29.018170 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:29.024371 8088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:29.024526 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:29.024562 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:29.024588 8088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:29.024609 8088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:29.032660 8088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:38565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:29.034368 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:29.035877 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:29.047186 8093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:29.047644 8088 server_base.cc:1034] running on GCE node
W20260923 18:10:29.047186 8094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:29.047200 8096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:29.048465 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:29.049751 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:29.051049 8088 hybrid_clock.cc:648] HybridClock initialized: now 1790187029051024 us; error 58 us; skew 500 ppm
Sep 23 18:10:29 dist-test-slave-rvjj krb5kdc[8075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187029, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:29.060772 8088 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:29.061976 8088 webserver.cc:460] Webserver started at http://127.18.44.254:45269/ using document root <none> and password file <none>
I20260923 18:10:29.062634 8088 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:29.062711 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:29.062960 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:29.065390 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "6fc51c38022249c5854f81447646f20e"
format_stamp: "Formatted at 2026-09-23 18:10:29 on dist-test-slave-rvjj"
server_key: "703cae5c4f6c5d6b1d23a407485c16a3"
server_key_iv: "8755c9de65955463ab770d2de1fd03f1"
server_key_version: "encryptionkey@0"
I20260923 18:10:29.066035 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "6fc51c38022249c5854f81447646f20e"
format_stamp: "Formatted at 2026-09-23 18:10:29 on dist-test-slave-rvjj"
server_key: "703cae5c4f6c5d6b1d23a407485c16a3"
server_key_iv: "8755c9de65955463ab770d2de1fd03f1"
server_key_version: "encryptionkey@0"
I20260923 18:10:29.070577 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:10:29.074297 8103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:29.076390 8088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:10:29.076581 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "6fc51c38022249c5854f81447646f20e"
format_stamp: "Formatted at 2026-09-23 18:10:29 on dist-test-slave-rvjj"
server_key: "703cae5c4f6c5d6b1d23a407485c16a3"
server_key_iv: "8755c9de65955463ab770d2de1fd03f1"
server_key_version: "encryptionkey@0"
I20260923 18:10:29.076714 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:29.098966 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:29.103456 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:29.103688 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:29.111990 8088 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:38565
I20260923 18:10:29.112114 8155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:38565 every 8 connection(s)
I20260923 18:10:29.113121 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:29.117213 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:29.122246 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8088
I20260923 18:10:29.122552 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:29.122987 18611 external_mini_cluster.cc:1428] Setting key 5a1684766546774137098e2d62763c89
I20260923 18:10:29.128014 8156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e: Bootstrap starting.
I20260923 18:10:29.131980 8156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:29.133133 8156 log.cc:824] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:29 dist-test-slave-rvjj krb5kdc[8075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:29.136888 8156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e: No bootstrap required, opened a new log
I20260923 18:10:29.142334 8156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } }
I20260923 18:10:29.142688 8156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:29.142750 8156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc51c38022249c5854f81447646f20e, State: Initialized, Role: FOLLOWER
I20260923 18:10:29.143440 8156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } }
I20260923 18:10:29.143502 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:29.125130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37993 (local address 127.18.44.254:38565)
0923 18:10:29.125698 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:29.125719 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:29.125770 (+ 51us) server_negotiation.cc:407] Connection header received
0923 18:10:29.126764 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:29.126804 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:29.127873 (+ 1069us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:29.128272 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:29.129561 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:29.130763 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:29.132374 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:29.133068 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:29.133263 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:29.137135 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:29.137143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:29.137164 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:29.137205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:29.139244 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:29.140121 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:29.140127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:29.140134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:29.140322 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:29.140732 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:29.140735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:29.140736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:29.141252 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:29.141471 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:29.141898 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 18:10:29.142255 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":183,"threads_started":1}
I20260923 18:10:29.143616 8156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:29.143687 8156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:29.143803 8156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:29.145138 8156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } }
I20260923 18:10:29.145588 8156 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fc51c38022249c5854f81447646f20e; no voters:
I20260923 18:10:29.146028 8156 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:29.146155 8161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:29.146543 8161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [term 1 LEADER]: Becoming Leader. State: Replica: 6fc51c38022249c5854f81447646f20e, State: Running, Role: LEADER
I20260923 18:10:29.147095 8161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } }
I20260923 18:10:29.147985 8156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:29.149277 8162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fc51c38022249c5854f81447646f20e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } } }
I20260923 18:10:29.149471 8162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:29.149781 8162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc51c38022249c5854f81447646f20e. Latest consensus state: current_term: 1 leader_uuid: "6fc51c38022249c5854f81447646f20e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc51c38022249c5854f81447646f20e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38565 } } }
I20260923 18:10:29.149864 8162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:29.150399 8170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:29.151358 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:29.155164 8170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:29.163465 8170 catalog_manager.cc:1347] Generated new cluster ID: d44f4fe1419c4053abe5ad5974b660c2
I20260923 18:10:29.163587 8170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:29.170392 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:29.158073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37855 (local address 127.18.44.254:38565)
0923 18:10:29.158201 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:29.158205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:29.162143 (+ 3938us) server_negotiation.cc:407] Connection header received
0923 18:10:29.162328 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:29.162332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:29.162413 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:29.162525 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:29.163597 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:29.164197 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:29.165198 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:29.165594 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:29.165723 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:29.167248 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:29.167276 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:29.167279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:29.167310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:29.168648 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:29.169060 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:29.169064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:29.169066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:29.169141 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:29.169438 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:29.169442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:29.169443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:29.169704 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:29.169821 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:29.170082 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 18:10:29.170219 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":45}
I20260923 18:10:29.219116 8170 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:29.220350 8170 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:29.229564 8170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc51c38022249c5854f81447646f20e: Generated new TSK 0
I20260923 18:10:29.230465 8170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:29.239708 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8088
2026-09-23T18:10:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:29 dist-test-slave-rvjj krb5kdc[8187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:29 dist-test-slave-rvjj krb5kdc[8187](info): set up 2 sockets
Sep 23 18:10:29 dist-test-slave-rvjj krb5kdc[8187](info): commencing operation
krb5kdc: starting...
W20260923 18:10:30.241083 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:30 dist-test-slave-rvjj krb5kdc[8187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:30.401674 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44905
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:44905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:30.541235 8200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:30.541553 8200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:30.541596 8200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:30.546464 8200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:30.546571 8200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:30.546591 8200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:30.546607 8200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:30.546622 8200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:30.553736 8200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:44905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:44905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:30.555048 8200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:30.556308 8200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:30.568019 8208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:30.568048 8205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:30.568378 8200 server_base.cc:1034] running on GCE node
W20260923 18:10:30.568029 8206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:30.569705 8200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:30.571686 8200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:30.573086 8200 hybrid_clock.cc:648] HybridClock initialized: now 1790187030573010 us; error 97 us; skew 500 ppm
Sep 23 18:10:30 dist-test-slave-rvjj krb5kdc[8187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187030, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:30.578213 8200 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:30.579891 8200 webserver.cc:460] Webserver started at http://127.18.44.254:38139/ using document root <none> and password file <none>
I20260923 18:10:30.580883 8200 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:30.581032 8200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:30.581346 8200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:30.585553 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "2a28f4ef189643299569e9d2e20d99f0"
format_stamp: "Formatted at 2026-09-23 18:10:30 on dist-test-slave-rvjj"
server_key: "ea1200815a90044d4da4dd5e9e8315b6"
server_key_iv: "952292cff50e77286d8e18857854f7b7"
server_key_version: "encryptionkey@0"
I20260923 18:10:30.586669 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "2a28f4ef189643299569e9d2e20d99f0"
format_stamp: "Formatted at 2026-09-23 18:10:30 on dist-test-slave-rvjj"
server_key: "ea1200815a90044d4da4dd5e9e8315b6"
server_key_iv: "952292cff50e77286d8e18857854f7b7"
server_key_version: "encryptionkey@0"
I20260923 18:10:30.592998 8200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 18:10:30.597198 8215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:30.599591 8200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:10:30.599831 8200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "2a28f4ef189643299569e9d2e20d99f0"
format_stamp: "Formatted at 2026-09-23 18:10:30 on dist-test-slave-rvjj"
server_key: "ea1200815a90044d4da4dd5e9e8315b6"
server_key_iv: "952292cff50e77286d8e18857854f7b7"
server_key_version: "encryptionkey@0"
I20260923 18:10:30.600042 8200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:30.633498 8200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:30.639484 8200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:30.639760 8200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:30.650022 8200 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:44905
I20260923 18:10:30.650111 8267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:44905 every 8 connection(s)
I20260923 18:10:30.651484 8200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:30.652163 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8200
I20260923 18:10:30.652446 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:30.652756 18611 external_mini_cluster.cc:1428] Setting key c0382aab70ba2e67678ef774b4a93f9c
I20260923 18:10:30.656019 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:30.667003 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:30.654323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37529 (local address 127.18.44.254:44905)
0923 18:10:30.654890 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:30.654908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:30.654952 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:10:30.655752 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:30.655777 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:30.656215 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:30.656603 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:30.657692 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:30.659712 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:30.661410 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:30.662492 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:30.662803 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:30.664236 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:30.664242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:30.664318 (+ 76us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:30.664817 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:30.664902 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:30.664934 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:30.665052 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:30.665520 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 18:10:30.665806 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":156,"threads_started":1}
I20260923 18:10:30.667773 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0: Bootstrap starting.
I20260923 18:10:30.671439 8268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:30.672293 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:30.672979 8268 log.cc:824] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:30.678360 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0: No bootstrap required, opened a new log
I20260923 18:10:30.680330 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:30.675818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49803 (local address 127.18.44.254:44905)
0923 18:10:30.675955 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:30.675960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:30.676207 (+ 247us) server_negotiation.cc:407] Connection header received
0923 18:10:30.676345 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:30.676348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:30.676397 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:30.676467 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:30.676782 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:30.677322 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:30.678243 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:30.678476 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:30.678571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:30.679831 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:30.679835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:30.679837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:30.679851 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:30.679884 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:30.679904 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:30.679972 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:30.680102 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 18:10:30.680171 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:10:30.685642 8268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } }
I20260923 18:10:30.686408 8268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:30.686523 8268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a28f4ef189643299569e9d2e20d99f0, State: Initialized, Role: FOLLOWER
I20260923 18:10:30.687477 8268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } }
I20260923 18:10:30.687718 8268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:30.687786 8268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:30.687901 8268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:30.690451 8268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } }
I20260923 18:10:30.691504 8268 leader_election.cc:302] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a28f4ef189643299569e9d2e20d99f0; no voters:
I20260923 18:10:30.692788 8268 leader_election.cc:288] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:30.693070 8278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:30.693665 8278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a28f4ef189643299569e9d2e20d99f0, State: Running, Role: LEADER
I20260923 18:10:30.694432 8278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } }
I20260923 18:10:30.695468 8268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:30.696801 8280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a28f4ef189643299569e9d2e20d99f0. Latest consensus state: current_term: 1 leader_uuid: "2a28f4ef189643299569e9d2e20d99f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } } }
I20260923 18:10:30.697012 8280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:30.696806 8279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a28f4ef189643299569e9d2e20d99f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f4ef189643299569e9d2e20d99f0" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 44905 } } }
I20260923 18:10:30.697319 8279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:30.697619 8287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:30.701432 8287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:30.712005 8287 catalog_manager.cc:1347] Generated new cluster ID: 3ca2448828174d9fbef744b980363f74
I20260923 18:10:30.712651 8287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:30.741604 8287 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:30.743592 8287 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:30.755785 8287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a28f4ef189643299569e9d2e20d99f0: Generated new TSK 0
I20260923 18:10:30.756731 8287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:30.772872 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8200
2026-09-23T18:10:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:30 dist-test-slave-rvjj krb5kdc[8299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:30 dist-test-slave-rvjj krb5kdc[8299](info): set up 2 sockets
Sep 23 18:10:30 dist-test-slave-rvjj krb5kdc[8299](info): commencing operation
krb5kdc: starting...
W20260923 18:10:31.796787 18611 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:31 dist-test-slave-rvjj krb5kdc[8299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:31.956066 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:38633
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:32.085994 8312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:32.086687 8312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:32.086781 8312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:32.092552 8312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:32.092666 8312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:32.092700 8312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:32.092726 8312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:32.092746 8312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:32.100482 8312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:32.101994 8312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:32.103440 8312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:32.114653 8318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:32.114636 8320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:32.114584 8317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:32.115041 8312 server_base.cc:1034] running on GCE node
I20260923 18:10:32.116163 8312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:32.118432 8312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:32.121941 8312 hybrid_clock.cc:648] HybridClock initialized: now 1790187032121830 us; error 117 us; skew 500 ppm
Sep 23 18:10:32 dist-test-slave-rvjj krb5kdc[8299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:32.127848 8312 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:32.129841 8312 webserver.cc:460] Webserver started at http://127.18.44.254:46605/ using document root <none> and password file <none>
I20260923 18:10:32.130734 8312 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:32.130847 8312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:32.131138 8312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:32.135684 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d"
format_stamp: "Formatted at 2026-09-23 18:10:32 on dist-test-slave-rvjj"
server_key: "87dbe9db26ca1a801e09cb3ccc96e40c"
server_key_iv: "f0de534d0e3a9912c87510150cab58c4"
server_key_version: "encryptionkey@0"
I20260923 18:10:32.136696 8312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d"
format_stamp: "Formatted at 2026-09-23 18:10:32 on dist-test-slave-rvjj"
server_key: "87dbe9db26ca1a801e09cb3ccc96e40c"
server_key_iv: "f0de534d0e3a9912c87510150cab58c4"
server_key_version: "encryptionkey@0"
I20260923 18:10:32.143452 8312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 18:10:32.148005 8327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:32.150169 8312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:32.150445 8312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d"
format_stamp: "Formatted at 2026-09-23 18:10:32 on dist-test-slave-rvjj"
server_key: "87dbe9db26ca1a801e09cb3ccc96e40c"
server_key_iv: "f0de534d0e3a9912c87510150cab58c4"
server_key_version: "encryptionkey@0"
I20260923 18:10:32.150632 8312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:32.157135 8312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:32.161875 8312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:32.162185 8312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:32.171909 8312 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:38633
I20260923 18:10:32.172174 8379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:38633 every 8 connection(s)
I20260923 18:10:32.173349 8312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:32.176432 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8312
I20260923 18:10:32.176604 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:32.176954 18611 external_mini_cluster.cc:1428] Setting key adf1c3f10ce030aa3423e116e6bcce26
I20260923 18:10:32.177306 8380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:32.184868 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:32.178936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60243 (local address 127.18.44.254:38633)
0923 18:10:32.179600 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:32.179621 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:32.179670 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:10:32.180511 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:32.180555 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:32.181113 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:32.181609 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:32.182927 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:32.182933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:32.182949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:32.182963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:32.183047 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:32.183073 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:32.183343 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:32.183516 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 18:10:32.183764 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":219,"threads_started":1}
I20260923 18:10:32.186206 8380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: Bootstrap starting.
I20260923 18:10:32.189172 8380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:32.189471 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:32.190328 8380 log.cc:824] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:32.193706 8380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: No bootstrap required, opened a new log
I20260923 18:10:32.195137 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:32.192109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50341 (local address 127.18.44.254:38633)
0923 18:10:32.192289 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:32.192295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:32.192506 (+ 211us) server_negotiation.cc:407] Connection header received
0923 18:10:32.192633 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:32.192637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:32.192681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:32.192938 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:32.194267 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:32.194271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:32.194275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:32.194290 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:32.194344 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:32.194361 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:32.194505 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:32.194945 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 18:10:32.194996 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 18:10:32.198048 8380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } }
I20260923 18:10:32.198347 8380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:32.198385 8380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e52f02e7fe44ba88b3acd16b50b6a6d, State: Initialized, Role: FOLLOWER
I20260923 18:10:32.198853 8380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } }
I20260923 18:10:32.199000 8380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:32.199054 8380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:32.199184 8380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:32.200304 8380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } }
I20260923 18:10:32.200670 8380 leader_election.cc:302] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e52f02e7fe44ba88b3acd16b50b6a6d; no voters:
I20260923 18:10:32.200978 8380 leader_election.cc:288] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:32.201282 8390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:32.201606 8390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [term 1 LEADER]: Becoming Leader. State: Replica: 4e52f02e7fe44ba88b3acd16b50b6a6d, State: Running, Role: LEADER
I20260923 18:10:32.202039 8390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } }
I20260923 18:10:32.202376 8380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:32.204811 8392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e52f02e7fe44ba88b3acd16b50b6a6d. Latest consensus state: current_term: 1 leader_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } } }
I20260923 18:10:32.205449 8392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:32.205060 8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e52f02e7fe44ba88b3acd16b50b6a6d" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 38633 } } }
I20260923 18:10:32.205891 8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d [sys.catalog]: This master's current role is: LEADER
W20260923 18:10:32.208112 8404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:32.208245 8404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:32.208397 8405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:32.209463 8405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:32.216807 8405 catalog_manager.cc:1347] Generated new cluster ID: e82f2017465f4bef871d2abad1f96dd1
I20260923 18:10:32.216939 8405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:32.233467 8405 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:32.234004 8405 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:32.244164 8405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e52f02e7fe44ba88b3acd16b50b6a6d: Generated new TSK 0
I20260923 18:10:32.245303 8405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:32.251680 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8312
2026-09-23T18:10:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1477 ms)
[----------] 18 tests from AuthTokenIssuingTest (26805 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:32 dist-test-slave-rvjj krb5kdc[8411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:32 dist-test-slave-rvjj krb5kdc[8411](info): set up 2 sockets
Sep 23 18:10:32 dist-test-slave-rvjj krb5kdc[8411](info): commencing operation
krb5kdc: starting...
W20260923 18:10:33.250126 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:33.425738 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:40403
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:33.551810 8424 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:33.552168 8424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:33.552204 8424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:33.552273 8424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:33.556932 8424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:33.557034 8424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:33.557062 8424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:33.557088 8424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:33.557109 8424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:33.563544 8424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:33.564756 8424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:33.565825 8424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:33.575644 8429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:33.575673 8430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:33.575703 8424 server_base.cc:1034] running on GCE node
W20260923 18:10:33.575644 8432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:33.576874 8424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:33.578533 8424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:33.579802 8424 hybrid_clock.cc:648] HybridClock initialized: now 1790187033579765 us; error 59 us; skew 500 ppm
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:33.583559 8424 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:33.584895 8424 webserver.cc:460] Webserver started at http://127.18.44.254:44953/ using document root <none> and password file <none>
I20260923 18:10:33.585625 8424 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:33.585702 8424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:33.585939 8424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:33.588792 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "d8e6c1544c4e49b1b5d237ea854a606e"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "fec0ff037d52a4b5ef81e82238c2cf05"
server_key_iv: "722d94b95d5e8bff8f64fe8fcdaa8940"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.589591 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "d8e6c1544c4e49b1b5d237ea854a606e"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "fec0ff037d52a4b5ef81e82238c2cf05"
server_key_iv: "722d94b95d5e8bff8f64fe8fcdaa8940"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.595182 8424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:10:33.598881 8439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:33.600908 8424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:33.601212 8424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "d8e6c1544c4e49b1b5d237ea854a606e"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "fec0ff037d52a4b5ef81e82238c2cf05"
server_key_iv: "722d94b95d5e8bff8f64fe8fcdaa8940"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.601377 8424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:33.617563 8424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:33.622916 8424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:33.623207 8424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:33.634119 8424 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:40403
I20260923 18:10:33.634418 8491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:40403 every 8 connection(s)
I20260923 18:10:33.635601 8424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:33.638949 8492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:33.643355 8492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:33.645790 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8424
I20260923 18:10:33.645939 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:33.646252 18611 external_mini_cluster.cc:1428] Setting key d4ead52957788e9fc5abc20812e8e52f
WARNING: no policy specified for kudu/127.18.44.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.253@KRBTEST.COM" created.
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:33.666987 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.646144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:40403 (local address 127.0.0.1:46566)
0923 18:10:33.646869 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.646935 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:33.648127 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:33.649038 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:33.649045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:33.649605 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:33.649912 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.649929 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.651624 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.651627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.652547 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.652558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.653338 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.653342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.653544 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.654252 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:33.654273 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:33.658907 (+ 4634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:33.662151 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.662170 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.662182 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.662599 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.663457 (+ 858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.663462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.663465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.663658 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.664878 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:33.664885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:33.665093 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 18:10:33.665305 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 18:10:33.665743 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":5,"thread_start_us":202,"threads_started":1}
I20260923 18:10:33.666990 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.646146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.18.44.254:40403)
0923 18:10:33.646853 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:33.646876 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:33.647094 (+ 218us) server_negotiation.cc:407] Connection header received
0923 18:10:33.648545 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:33.648551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:33.648811 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:33.649012 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:33.650099 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.651468 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.652693 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.653147 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.653328 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.659354 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:33.659368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:33.659396 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:33.659449 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:33.661872 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.662869 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.662880 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.662882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.663241 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.663831 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.663836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.663838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.664666 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:33.664856 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:33.665515 (+ 659us) server_negotiation.cc:299] Negotiation successful
0923 18:10:33.665875 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":41216,"thread_start_us":209,"threads_started":1}
I20260923 18:10:33.670621 8492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:33.671787 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.671441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54328)
0923 18:10:33.671586 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.671689 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 18:10:33.672036 8441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
I20260923 18:10:33.673394 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.673017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54336)
0923 18:10:33.673179 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.673321 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260923 18:10:33.673568 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:36855: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
I20260923 18:10:33.685081 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:36855
--webserver_interface=127.18.44.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:10:33.722852 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } attempt: 1
I20260923 18:10:33.724045 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.723511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54350)
0923 18:10:33.723779 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.723901 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 18:10:33.725245 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.724938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54364)
0923 18:10:33.725081 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.725154 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 18:10:33.725500 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:36855: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
I20260923 18:10:33.793776 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } attempt: 2
I20260923 18:10:33.794960 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.794370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54370)
0923 18:10:33.794707 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.794821 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 18:10:33.796126 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.795825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54384)
0923 18:10:33.795955 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.796031 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 18:10:33.796370 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:36855: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
W20260923 18:10:33.835455 8499 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:33.835906 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:33.835958 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:33.836026 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:33.840870 8499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:33.840997 8499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:33.841030 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:33.841054 8499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:33.841075 8499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:33.848227 8499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:36855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.44.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:33.850333 8499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:33.851950 8499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:33.864037 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:33.864040 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:33.864040 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:33.864682 8499 server_base.cc:1034] running on GCE node
I20260923 18:10:33.865413 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:33.866732 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:33.867955 8499 hybrid_clock.cc:648] HybridClock initialized: now 1790187033867922 us; error 76 us; skew 500 ppm
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:33.871518 8499 init.cc:377] Logged in from keytab as kudu/127.18.44.253@KRBTEST.COM (short username kudu)
I20260923 18:10:33.872710 8499 webserver.cc:460] Webserver started at http://127.18.44.253:44647/ using document root <none> and password file <none>
I20260923 18:10:33.873481 8499 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:33.873567 8499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:33.873821 8499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:33.876695 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data/instance:
uuid: "bfebd5a5bd514e5083540cad03809932"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "6518914b516d084fadfd273450efaddb"
server_key_iv: "d58b1dc13304d400875bb83e7510cc98"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.877455 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal/instance:
uuid: "bfebd5a5bd514e5083540cad03809932"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "6518914b516d084fadfd273450efaddb"
server_key_iv: "d58b1dc13304d400875bb83e7510cc98"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.882433 8499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:10:33.886004 8514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:33.887580 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } attempt: 3
I20260923 18:10:33.887974 8499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 18:10:33.888128 8499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "bfebd5a5bd514e5083540cad03809932"
format_stamp: "Formatted at 2026-09-23 18:10:33 on dist-test-slave-rvjj"
server_key: "6518914b516d084fadfd273450efaddb"
server_key_iv: "d58b1dc13304d400875bb83e7510cc98"
server_key_version: "encryptionkey@0"
I20260923 18:10:33.888262 8499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:33.888823 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.888337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54388)
0923 18:10:33.888495 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.888714 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 18:10:33.889654 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.889406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54402)
0923 18:10:33.889522 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.889575 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 18:10:33.889839 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:36855: Network error: Client connection negotiation failed: client connection to 127.18.44.253:36855: connect: Connection refused (error 111)
I20260923 18:10:33.914295 8499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:33.918118 8499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:33.918303 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:33.926513 8499 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.253:36855
I20260923 18:10:33.926889 8566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.253:36855 every 8 connection(s)
I20260923 18:10:33.927763 8499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
I20260923 18:10:33.932477 8567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:33.935930 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8499
I20260923 18:10:33.936065 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-1/wal/instance
I20260923 18:10:33.936333 18611 external_mini_cluster.cc:1428] Setting key 4f32bb617b47226587d70d1e7ac587f1
I20260923 18:10:33.938334 8567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.44.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.252@KRBTEST.COM" created.
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:33.957024 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.941438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.18.44.254:40403)
0923 18:10:33.941674 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:33.941680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:33.942741 (+ 1061us) server_negotiation.cc:407] Connection header received
0923 18:10:33.943905 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:33.943910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:33.944004 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:33.944163 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:33.945593 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.946441 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.948002 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.948308 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.948441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.951853 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:33.951857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:33.951860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:33.951897 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:33.954008 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.954926 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.954930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.954931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.955036 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.955524 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.955527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.955529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.955776 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:33.955872 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:33.956642 (+ 770us) server_negotiation.cc:299] Negotiation successful
0923 18:10:33.956821 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 18:10:33.958194 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.941902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:40403 (local address 127.0.0.1:46574)
0923 18:10:33.942493 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.942559 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:33.943646 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:33.944378 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:33.944385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:33.944837 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:33.945364 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.945379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.946646 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.946649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.947812 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.947825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.948529 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.948533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.948719 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.949425 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:33.949445 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:33.951542 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:33.954290 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.954299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.954323 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.954766 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.955153 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.955156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.955158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.955410 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.955898 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:33.955918 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:33.956315 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 18:10:33.956531 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 18:10:33.957014 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":176,"threads_started":1}
I20260923 18:10:33.960657 8567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:10:33 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
I20260923 18:10:33.973374 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:45961
--webserver_interface=127.18.44.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:10:33.974126 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.961460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54404)
0923 18:10:33.961632 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.961649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:33.961761 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:33.962608 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:33.962619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:33.962914 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:33.963130 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.963141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.964723 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.964727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.965464 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:33.965478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.966492 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.966496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:33.966606 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.967184 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:33.967196 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:33.968859 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:33.971538 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.971543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.971545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.971866 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.972481 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:33.972485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:33.972488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:33.972661 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:33.973380 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:33.973383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:33.973574 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 18:10:33.973715 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 18:10:33.973923 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 18:10:33.974817 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.961599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.18.44.253:36855)
0923 18:10:33.962052 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:33.962066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:33.962088 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:10:33.962136 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:33.962141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:33.962380 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:33.962520 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:33.963384 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.964521 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.965655 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:33.966314 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:33.966436 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:33.969121 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:33.969140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:33.969172 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:33.969200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:33.971309 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.972064 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.972072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.972073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.972304 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:33.972786 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:33.972790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:33.972792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:33.973219 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:33.973391 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:33.974468 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0923 18:10:33.974680 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260923 18:10:33.977617 8567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:33.978776 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.978280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33608)
0923 18:10:33.978526 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.978668 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260923 18:10:33.979045 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:33.980330 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:33.979988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33612)
0923 18:10:33.980152 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:33.980232 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 18:10:33.980545 8567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.029983 8567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 1
I20260923 18:10:34.031414 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.030808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33616)
0923 18:10:34.031082 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.031205 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 18:10:34.032481 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.032144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33626)
0923 18:10:34.032318 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.032398 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 18:10:34.032693 8567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.033064 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } attempt: 4
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:10:34.046695 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.033736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54414)
0923 18:10:34.033922 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.033939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.034056 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.034552 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.034555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.034759 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.034924 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.034931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.036387 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.036391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.037696 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.037716 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.038533 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.038536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.038633 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.039361 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.039375 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.041075 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.044527 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.044531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.044533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.044870 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.045387 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.045390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.045392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.045508 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.046155 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.046158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.046288 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.046390 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.046553 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 18:10:34.046862 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.033794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.18.44.253:36855)
0923 18:10:34.033936 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.033941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.034022 (+ 81us) server_negotiation.cc:407] Connection header received
0923 18:10:34.034227 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.034232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.034314 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.034515 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.035119 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.036085 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.037952 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.038369 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.038503 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.041351 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.041356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.041360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.041412 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.044297 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.045066 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.045071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.045074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.045222 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.045665 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.045670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.045672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.045993 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.046147 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.046484 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.046669 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:10:34.048113 8492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:34.048950 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.048684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33636)
0923 18:10:34.048819 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.048879 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 18:10:34.049974 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.049676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33638)
0923 18:10:34.049846 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.049914 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260923 18:10:34.050163 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.100924 8567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 2
I20260923 18:10:34.102171 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.101669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33644)
0923 18:10:34.101900 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.102015 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 18:10:34.103209 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.102840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33654)
0923 18:10:34.103033 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.103114 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 18:10:34.103529 8567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.109347 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 1
I20260923 18:10:34.110235 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.109894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33666)
0923 18:10:34.110038 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.110095 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 18:10:34.111462 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.111003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33668)
0923 18:10:34.111353 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.111411 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260923 18:10:34.111625 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
W20260923 18:10:34.136799 8573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:34.137265 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:34.137293 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:34.137341 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:34.142230 8573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:34.142359 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:34.142379 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:34.142396 8573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:34.142412 8573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:34.149588 8573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:40403,127.18.44.253:36855,127.18.44.252:45961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.44.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:34.151206 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:34.152827 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:34.164945 8582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:34.164945 8579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:34.164976 8580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:34.165567 8573 server_base.cc:1034] running on GCE node
I20260923 18:10:34.166260 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:34.167625 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:34.170989 8573 hybrid_clock.cc:648] HybridClock initialized: now 1790187034171051 us; error 136 us; skew 500 ppm
I20260923 18:10:34.178843 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 2
I20260923 18:10:34.180107 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.179714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33676)
0923 18:10:34.179933 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.180010 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187034, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:34.180832 8573 init.cc:377] Logged in from keytab as kudu/127.18.44.252@KRBTEST.COM (short username kudu)
I20260923 18:10:34.181105 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.180824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33682)
0923 18:10:34.180959 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.181038 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 18:10:34.181453 8492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.182665 8573 webserver.cc:460] Webserver started at http://127.18.44.252:37327/ using document root <none> and password file <none>
I20260923 18:10:34.183580 8573 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:34.183681 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:34.183961 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:34.186944 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data/instance:
uuid: "050e8f33771d4e7e933121595e47ee7a"
format_stamp: "Formatted at 2026-09-23 18:10:34 on dist-test-slave-rvjj"
server_key: "9dd3e9e7696cb16ea8d5cf370612972d"
server_key_iv: "fe36528d64eff79ab9e7ea41d29cfce6"
server_key_version: "encryptionkey@0"
I20260923 18:10:34.187868 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal/instance:
uuid: "050e8f33771d4e7e933121595e47ee7a"
format_stamp: "Formatted at 2026-09-23 18:10:34 on dist-test-slave-rvjj"
server_key: "9dd3e9e7696cb16ea8d5cf370612972d"
server_key_iv: "fe36528d64eff79ab9e7ea41d29cfce6"
server_key_version: "encryptionkey@0"
I20260923 18:10:34.193591 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:34.194734 8567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 3
I20260923 18:10:34.196003 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.195569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33696)
0923 18:10:34.195786 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.195887 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 18:10:34.197089 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.196674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33702)
0923 18:10:34.196915 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.196990 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260923 18:10:34.197310 8567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:45961: Network error: Client connection negotiation failed: client connection to 127.18.44.252:45961: connect: Connection refused (error 111)
I20260923 18:10:34.198287 8589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:34.200753 8573 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:10:34.201148 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "050e8f33771d4e7e933121595e47ee7a"
format_stamp: "Formatted at 2026-09-23 18:10:34 on dist-test-slave-rvjj"
server_key: "9dd3e9e7696cb16ea8d5cf370612972d"
server_key_iv: "fe36528d64eff79ab9e7ea41d29cfce6"
server_key_version: "encryptionkey@0"
I20260923 18:10:34.201339 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:34.220544 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:34.226334 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:34.226626 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:34.238224 8573 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.252:45961
I20260923 18:10:34.238330 8641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.252:45961 every 8 connection(s)
I20260923 18:10:34.239930 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
I20260923 18:10:34.243983 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:34.245155 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8573
I20260923 18:10:34.245304 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790186727112038-18611-0/minicluster-data/master-2/wal/instance
I20260923 18:10:34.245625 18611 external_mini_cluster.cc:1428] Setting key b7f9c3cd43469b4482ffe51d2c38bd07
I20260923 18:10:34.250087 8642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187034, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:34.267915 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.253502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.18.44.254:40403)
0923 18:10:34.253643 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.253646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.254332 (+ 686us) server_negotiation.cc:407] Connection header received
0923 18:10:34.254512 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.254516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.254567 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.254636 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.255671 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.256427 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.257791 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.258136 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.258269 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.262498 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.262503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.262505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.262547 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.264521 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.265433 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.265437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.265439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.265532 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.266121 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.266125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.266127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.266470 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.266584 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.267522 (+ 938us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.267761 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:10:34.269263 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.248188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.18.44.252:45961)
0923 18:10:34.248935 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.248979 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.249040 (+ 61us) server_negotiation.cc:407] Connection header received
0923 18:10:34.250002 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.250033 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.250568 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.251299 (+ 731us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.252874 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.255205 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.256393 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.257542 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.257792 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.261663 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.261672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.261687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.261725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.264281 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.265264 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.265350 (+ 86us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.265352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.265575 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.266121 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.266125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.266127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.266796 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.266986 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.267606 (+ 620us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.268022 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":301,"server-negotiator.queue_time_us":524,"thread_start_us":251,"threads_started":1}
I20260923 18:10:34.269261 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.253462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:40403 (local address 127.0.0.1:46588)
0923 18:10:34.254133 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.254204 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.254394 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.254736 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.254743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.255207 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.255452 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.255464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.256670 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.256674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.257578 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.257592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.258460 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.258466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.258601 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.259602 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.259652 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.262117 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.264693 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.264712 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.264731 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.265227 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.265820 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.265824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.265826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.266000 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.266702 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.266711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.266997 (+ 286us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.267369 (+ 372us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.267929 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":6,"thread_start_us":276,"threads_started":1}
I20260923 18:10:34.271546 8642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187034, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:34.281632 8492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 3
I20260923 18:10:34.284030 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.272186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:36855 (local address 127.0.0.1:54424)
0923 18:10:34.272339 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.272357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.272455 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.272948 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.272952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.273243 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.273398 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.273406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.274774 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.274776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.275441 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.275454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.275976 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.275980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.276107 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.276835 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.276849 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.279162 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.281815 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.281822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.281824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.282186 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.282677 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.282680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.282682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.282766 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.283312 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.283315 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.283429 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.283562 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.283737 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 18:10:34.284036 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.272439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.18.44.253:36855)
0923 18:10:34.272619 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.272624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.272644 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:34.272710 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.272714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.272777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.272871 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.273546 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.274556 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.275592 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.275836 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.275944 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.279510 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.279516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.279519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.279563 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.281505 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.282461 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.282466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.282467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.282576 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.282870 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.282873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.282875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.283126 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.283228 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.283648 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.283859 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:10:34.285490 8642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:34.285797 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.272601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.18.44.254:40403)
0923 18:10:34.272723 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.272728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.272785 (+ 57us) server_negotiation.cc:407] Connection header received
0923 18:10:34.272929 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.272933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.272984 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.273070 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.274009 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.274728 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.275707 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.275936 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.276035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.280627 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.280631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.280633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.280668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.283038 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.283963 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.283968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.283970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.284194 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.284562 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.284566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.284567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.284861 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.285337 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.285440 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.285603 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187034, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:10:34.293380 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.282116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33720)
0923 18:10:34.282332 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.282346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.282436 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.283094 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.283097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.283299 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.283424 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.283431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.284978 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.284983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.285813 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.285835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.286545 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.286550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.286675 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.287703 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.287718 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.289152 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.291321 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.291325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.291329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.291686 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.292147 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.292151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.292153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.292293 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.292811 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.292815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.292951 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.293045 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.293202 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 18:10:34.293988 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.282208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.18.44.252:45961)
0923 18:10:34.282332 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.282336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.282390 (+ 54us) server_negotiation.cc:407] Connection header received
0923 18:10:34.282524 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.282527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.282581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.283087 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.283632 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.284570 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.286006 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.286384 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.286942 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.289348 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.289353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.289356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.289398 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.291155 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.291817 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.291821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.291823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.291946 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.292445 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.292448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.292450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.292690 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.293581 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.293679 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.293825 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:10:34.298231 8492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: Bootstrap starting.
I20260923 18:10:34.298744 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.286221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33722)
0923 18:10:34.286508 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.286525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.286608 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.287477 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.287480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.287668 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.287805 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.287813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.289447 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.289451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.290278 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.290292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.291052 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.291055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.291165 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.291935 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.291949 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.293688 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.296381 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.296385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.296387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.296692 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.297061 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.297065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.297067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.297176 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.298093 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.298096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.298179 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.298306 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.298565 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"spinlock_wait_cycles":6400}
I20260923 18:10:34.299125 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.286521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.18.44.252:45961)
0923 18:10:34.286953 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.286964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.286982 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:10:34.287031 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.287034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.287093 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.287440 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.287961 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.289294 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.290438 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.290877 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.291028 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.293825 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.293828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.293831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.293850 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.296201 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.296825 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.296828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.296830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.296932 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.297613 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.297617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.297619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.297949 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.298057 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.298499 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.298703 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":181,"threads_started":1}
I20260923 18:10:34.301467 8492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:34.301934 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.289227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.18.44.253:36855)
0923 18:10:34.289665 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.289669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.289683 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:10:34.289727 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.289730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.289796 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.289870 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.290759 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.291403 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.292617 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.293017 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.293118 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.296879 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.296884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.296886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.296936 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.299222 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.300059 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.300063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.300065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.300185 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.300730 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.300736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.300737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.301069 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.301361 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.301552 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.301713 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260923 18:10:34.302618 8492 log.cc:824] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:34.304077 8642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: Bootstrap starting.
I20260923 18:10:34.307144 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:34.307350 8492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: No bootstrap required, opened a new log
I20260923 18:10:34.309181 8642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:34.308772 8466 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d8e6c1544c4e49b1b5d237ea854a606e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46596
I20260923 18:10:34.310860 8642 log.cc:824] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:34.312682 8492 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.313069 8492 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e6c1544c4e49b1b5d237ea854a606e, State: Initialized, Role: FOLLOWER
I20260923 18:10:34.313525 8492 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.314330 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.314509 8642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: No bootstrap required, opened a new log
I20260923 18:10:34.314567 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:34.315596 8492 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:34.315724 8466 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 18:10:34.315892 8466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:34.317265 8466 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.319629 8466 leader_election.cc:288] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [CANDIDATE]: Term 1 election: Requested vote from peers bfebd5a5bd514e5083540cad03809932 (127.18.44.253:36855), 050e8f33771d4e7e933121595e47ee7a (127.18.44.252:45961)
I20260923 18:10:34.320061 8642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.320638 8642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050e8f33771d4e7e933121595e47ee7a, State: Initialized, Role: FOLLOWER
I20260923 18:10:34.321828 8642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.324204 8642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:10:34.324349 8661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:34.324438 8661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:34.324304 8613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "050e8f33771d4e7e933121595e47ee7a"
I20260923 18:10:34.324692 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:34.326905 8613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e6c1544c4e49b1b5d237ea854a606e in term 1.
I20260923 18:10:34.327612 8442 leader_election.cc:302] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050e8f33771d4e7e933121595e47ee7a, d8e6c1544c4e49b1b5d237ea854a606e; no voters:
I20260923 18:10:34.328276 8650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:34.328773 8650 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [term 1 LEADER]: Becoming Leader. State: Replica: d8e6c1544c4e49b1b5d237ea854a606e, State: Running, Role: LEADER
I20260923 18:10:34.329218 8650 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.324266 8424 master_runner.cc:186] Error getting master registration for 127.18.44.253:36855: OK, instance_id { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" instance_seqno: 1790187033923968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:10:34.330147 8424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:10:34.333837 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e6c1544c4e49b1b5d237ea854a606e. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.334189 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:34.335489 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.335664 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: This master's current role is: FOLLOWER
W20260923 18:10:34.340359 8680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:34.340539 8680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:34.340466 8567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } attempt: 4
I20260923 18:10:34.338343 8573 master_runner.cc:186] Error getting master registration for 127.18.44.253:36855: OK, instance_id { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" instance_seqno: 1790187033923968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:10:34.342672 8573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:10:34.347448 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.325479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.18.44.253:36855)
0923 18:10:34.325632 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.325637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.330756 (+ 5119us) server_negotiation.cc:407] Connection header received
0923 18:10:34.335582 (+ 4826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.335587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.335666 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.335729 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.336916 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.337694 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.338869 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.339217 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.339341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.341918 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.341921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.341924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.341947 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.344725 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.345512 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.345516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.345518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.345628 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.346072 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.346075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.346077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.346355 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.346497 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.346940 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.347101 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":7040}
Sep 23 18:10:34 dist-test-slave-rvjj krb5kdc[8411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:10:34.354357 8685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:34.356616 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.325985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.18.44.252:45961)
0923 18:10:34.326234 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.326240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.328006 (+ 1766us) server_negotiation.cc:407] Connection header received
0923 18:10:34.328193 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.328197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.328271 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.328390 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.330515 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.331399 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.334750 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.335105 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.335203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.341624 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.341631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.341634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.341674 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.343974 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.347620 (+ 3646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.347625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.347627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.347761 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.349013 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.349019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.349021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.349349 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.349813 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.355648 (+ 5835us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.355898 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260923 18:10:34.356841 8685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:34.358289 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.341272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:45961 (local address 127.0.0.1:33730)
0923 18:10:34.341677 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:34.341695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:34.341788 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:34.343038 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:34.343042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:34.343464 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:34.343624 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.343633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.344768 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.344773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.345640 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:34.345656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.346721 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.346726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:34.346837 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.347465 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:34.347481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:34.350977 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:34.355752 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.355756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.355759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.356100 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.356435 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:34.356438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:34.356440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:34.356528 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:34.357618 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:34.357622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:34.357783 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 18:10:34.357895 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:10:34.358093 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"spinlock_wait_cycles":7424}
I20260923 18:10:34.359100 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.342374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.18.44.252:45961)
0923 18:10:34.342543 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.342548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.342564 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:10:34.342632 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.342635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.342691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.342822 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.343813 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.344598 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.345870 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.346369 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.346499 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.351489 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.351494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.351497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.351531 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.355533 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.356233 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.356237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.356239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.356337 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.356626 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.356630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.356632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.357318 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.357554 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.358815 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.358959 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 18:10:34.360333 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:34.327424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.18.44.254:40403)
0923 18:10:34.327586 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.327592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.328746 (+ 1154us) server_negotiation.cc:407] Connection header received
0923 18:10:34.328946 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.328949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.329003 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.329053 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.332753 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.333500 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.336204 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:34.336600 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:34.336692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:34.344823 (+ 8131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:34.344829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:34.344832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:34.344877 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:34.346855 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.347921 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.347925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.347927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.348039 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:34.350348 (+ 2309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:34.350354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:34.350356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:34.351382 (+ 1026us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:34.352481 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:34.359796 (+ 7315us) server_negotiation.cc:299] Negotiation successful
0923 18:10:34.360125 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 18:10:34.364318 8567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: Bootstrap starting.
I20260923 18:10:34.368163 8567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:34.368790 8613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [term 1 FOLLOWER]: Refusing update from remote peer d8e6c1544c4e49b1b5d237ea854a606e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:34.369791 8567 log.cc:824] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:34.369834 8682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [LEADER]: Connected to new peer: Peer: permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:34.374975 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e6c1544c4e49b1b5d237ea854a606e. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.375180 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: This master's current role is: FOLLOWER
W20260923 18:10:34.376348 8445 tablet.cc:2369] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:10:34.379199 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.379382 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:34.379508 8686 mvcc.cc:205] Tried to move back new op lower bound from 7332606092763201536 to 7332606092616601600. Current Snapshot: MvccSnapshot[applied={T|T < 7332606092763201536}]
I20260923 18:10:34.381228 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.381371 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:34.381798 8567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: No bootstrap required, opened a new log
I20260923 18:10:34.384037 8685 catalog_manager.cc:1347] Generated new cluster ID: f3912e9128ce4df18ddf510be12c860b
I20260923 18:10:34.384126 8685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:34.394306 8567 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.394693 8567 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfebd5a5bd514e5083540cad03809932, State: Initialized, Role: FOLLOWER
I20260923 18:10:34.395644 8688 mvcc.cc:205] Tried to move back new op lower bound from 7332606092763201536 to 7332606092616601600. Current Snapshot: MvccSnapshot[applied={T|T < 7332606092763201536}]
I20260923 18:10:34.400085 8567 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } }
I20260923 18:10:34.402086 8567 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:34.402390 8541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:34.403345 8539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bfebd5a5bd514e5083540cad03809932"
I20260923 18:10:34.404762 8541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [term 1 FOLLOWER]: Refusing update from remote peer d8e6c1544c4e49b1b5d237ea854a606e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:34.405608 8650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:34.407948 8539 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e6c1544c4e49b1b5d237ea854a606e in term 1.
I20260923 18:10:34.417761 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.417996 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:34.418121 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e6c1544c4e49b1b5d237ea854a606e. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.418198 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:34.418426 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.418511 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 18:10:34.420159 8520 tablet.cc:2369] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:34.422472 8703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:34.423017 8703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:34.425412 8705 mvcc.cc:205] Tried to move back new op lower bound from 7332606092763201536 to 7332606092616601600. Current Snapshot: MvccSnapshot[applied={T|T < 7332606092763201536}]
I20260923 18:10:34.437999 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e6c1544c4e49b1b5d237ea854a606e" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 40403 } } peers { permanent_uuid: "bfebd5a5bd514e5083540cad03809932" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 36855 } } peers { permanent_uuid: "050e8f33771d4e7e933121595e47ee7a" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 45961 } } }
I20260923 18:10:34.438164 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:34.444451 8685 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:34.445544 8685 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:34.461019 8685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e6c1544c4e49b1b5d237ea854a606e: Generated new TSK 0
I20260923 18:10:34.462196 8685 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:10:34.498276 8645 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.494870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.18.44.252:45961)
0923 18:10:34.495011 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.495016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.497061 (+ 2045us) server_negotiation.cc:407] Connection header received
0923 18:10:34.497288 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.497292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.497346 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.497399 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.498130 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33744: BlockingRecv error: recv got EOF from 127.0.0.1:33744 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 18:10:34.499818 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.495799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.18.44.253:36855)
0923 18:10:34.495920 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.495924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.498705 (+ 2781us) server_negotiation.cc:407] Connection header received
0923 18:10:34.498894 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.498897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.498955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.499008 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.499660 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54456: BlockingRecv error: recv got EOF from 127.0.0.1:54456 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 18:10:34.501667 8495 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.496521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.18.44.254:40403)
0923 18:10:34.496642 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.496646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.500412 (+ 3766us) server_negotiation.cc:407] Connection header received
0923 18:10:34.500593 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.500596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.500652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.500845 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.501518 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46600: BlockingRecv error: recv got EOF from 127.0.0.1:46600 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 18:10:34.507771 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.505718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.18.44.253:36855)
0923 18:10:34.505841 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.505846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.506542 (+ 696us) server_negotiation.cc:407] Connection header received
0923 18:10:34.506804 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.506808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.506900 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.506950 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.507631 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54458: BlockingRecv error: recv got EOF from 127.0.0.1:54458 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 18:10:34.513484 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.511741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.18.44.253:36855)
0923 18:10:34.511869 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.511873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.512450 (+ 577us) server_negotiation.cc:407] Connection header received
0923 18:10:34.512655 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.512658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.512712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.512765 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.513358 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54466: BlockingRecv error: recv got EOF from 127.0.0.1:54466 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 18:10:34.520949 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.518572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.18.44.253:36855)
0923 18:10:34.518854 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.518859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.519661 (+ 802us) server_negotiation.cc:407] Connection header received
0923 18:10:34.520000 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.520004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.520066 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.520124 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.520813 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54480: BlockingRecv error: recv got EOF from 127.0.0.1:54480 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
W20260923 18:10:34.531855 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.529307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.18.44.253:36855)
0923 18:10:34.529506 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.529512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.530563 (+ 1051us) server_negotiation.cc:407] Connection header received
0923 18:10:34.530825 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.530829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.530896 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.530959 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.531714 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54490: BlockingRecv error: recv got EOF from 127.0.0.1:54490 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 18:10:34.546383 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.543945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.18.44.253:36855)
0923 18:10:34.544086 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.544091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.545028 (+ 937us) server_negotiation.cc:407] Connection header received
0923 18:10:34.545319 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.545324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.545393 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.545456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.546239 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54498: BlockingRecv error: recv got EOF from 127.0.0.1:54498 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 18:10:34.569438 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.567318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.18.44.253:36855)
0923 18:10:34.567488 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.567493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.568295 (+ 802us) server_negotiation.cc:407] Connection header received
0923 18:10:34.568537 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.568541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.568602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.568659 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.569304 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54514: BlockingRecv error: recv got EOF from 127.0.0.1:54514 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 18:10:34.605907 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.604583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.18.44.253:36855)
0923 18:10:34.604715 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.604721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.604947 (+ 226us) server_negotiation.cc:407] Connection header received
0923 18:10:34.605143 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.605147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.605234 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.605319 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.605784 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54526: BlockingRecv error: recv got EOF from 127.0.0.1:54526 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 18:10:34.673378 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.672134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.18.44.253:36855)
0923 18:10:34.672328 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.672332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.672400 (+ 68us) server_negotiation.cc:407] Connection header received
0923 18:10:34.672520 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.672523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.672575 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.672678 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.673259 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54534: BlockingRecv error: recv got EOF from 127.0.0.1:54534 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260923 18:10:34.805254 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:34.803768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.18.44.253:36855)
0923 18:10:34.803980 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:34.803985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:34.804272 (+ 287us) server_negotiation.cc:407] Connection header received
0923 18:10:34.804450 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:34.804453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:34.804506 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:34.804591 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:34.805128 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54540: BlockingRecv error: recv got EOF from 127.0.0.1:54540 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 18:10:35.064723 8572 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:35.063567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.18.44.253:36855)
0923 18:10:35.063724 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.063729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.063832 (+ 103us) server_negotiation.cc:407] Connection header received
0923 18:10:35.063984 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.063988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.064035 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.064106 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:35.064605 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54542: BlockingRecv error: recv got EOF from 127.0.0.1:54542 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 18:10:35.342024 8680 catalog_manager.cc:1259] Loaded cluster ID: f3912e9128ce4df18ddf510be12c860b
I20260923 18:10:35.342124 8680 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: loading cluster ID for follower catalog manager: success
I20260923 18:10:35.343894 8680 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: acquiring CA information for follower catalog manager: success
I20260923 18:10:35.345160 8680 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 050e8f33771d4e7e933121595e47ee7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:10:35.425143 8703 catalog_manager.cc:1259] Loaded cluster ID: f3912e9128ce4df18ddf510be12c860b
I20260923 18:10:35.425284 8703 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: loading cluster ID for follower catalog manager: success
I20260923 18:10:35.428002 8703 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: acquiring CA information for follower catalog manager: success
I20260923 18:10:35.429773 8703 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfebd5a5bd514e5083540cad03809932: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:10:35.585278 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:35.579621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.18.44.253:36855)
0923 18:10:35.579925 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.579929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.579949 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:35.580165 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.580168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.580311 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.580463 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:35.580967 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.581652 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.583131 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.583464 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.583596 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:35.584108 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:35.584697 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:35.584817 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:35.585050 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 18:10:35.585127 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260923 18:10:35.595400 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:35.589937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.18.44.252:45961)
0923 18:10:35.590418 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.590422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.590442 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:35.590594 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.590597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.590730 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.590868 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:35.591294 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.591960 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.593359 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.593612 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.593706 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:35.594304 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:35.594788 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:35.594921 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:35.595139 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 18:10:35.595205 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":143,"threads_started":1}
I20260923 18:10:35.604194 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:35.599888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.18.44.253:36855)
0923 18:10:35.600003 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.600007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.600024 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:10:35.600087 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.600091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.600150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.600254 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:35.600678 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.601584 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.602764 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.603063 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.603179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:35.603491 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:35.603603 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:35.603672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:35.603812 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 18:10:35.603888 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:10:35.604633 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:35.599829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.18.44.252:45961)
0923 18:10:35.599966 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.599970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.600270 (+ 300us) server_negotiation.cc:407] Connection header received
0923 18:10:35.600453 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.600456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.600504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.600565 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:35.600940 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.601709 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.603113 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.603515 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.603606 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:35.603900 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:35.603997 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:35.604110 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:35.604306 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 18:10:35.604495 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:10:35.605849 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:35.599400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.18.44.254:40403)
0923 18:10:35.600162 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:35.600166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:35.600179 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:10:35.600313 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:35.600317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:35.600534 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:35.600713 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:35.600990 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.601568 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.603106 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:35.603545 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:35.603700 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:35.604218 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:35.604927 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:35.605035 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:35.605256 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 18:10:35.605691 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":99,"threads_started":1}
I20260923 18:10:35.610620 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8424
I20260923 18:10:35.619053 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8499
I20260923 18:10:35.626708 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8573
2026-09-23T18:10:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3376 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:35 dist-test-slave-rvjj krb5kdc[8792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:35 dist-test-slave-rvjj krb5kdc[8792](info): set up 2 sockets
Sep 23 18:10:35 dist-test-slave-rvjj krb5kdc[8792](info): commencing operation
krb5kdc: starting...
W20260923 18:10:36.613328 18611 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/test-user.keytab.
Sep 23 18:10:36 dist-test-slave-rvjj krb5kdc[8792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.44.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
Entry for principal kudu/127.18.44.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab.
I20260923 18:10:36.777289 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46657
--webserver_interface=127.18.44.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:36.904577 8805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:36.904914 8805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:36.904948 8805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:36.905014 8805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:36.909247 8805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:36.909340 8805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:36.909368 8805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:36.909387 8805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:36.909405 8805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:36.915071 8805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.254.keytab
--principal=kudu/127.18.44.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.254:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.44.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:36.916361 8805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:36.917455 8805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:36.928637 8813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:36.928663 8810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:36.928653 8811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:36.928788 8805 server_base.cc:1034] running on GCE node
I20260923 18:10:36.929899 8805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:36.931293 8805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:36.932580 8805 hybrid_clock.cc:648] HybridClock initialized: now 1790187036932542 us; error 73 us; skew 500 ppm
Sep 23 18:10:36 dist-test-slave-rvjj krb5kdc[8792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:36.936124 8805 init.cc:377] Logged in from keytab as kudu/127.18.44.254@KRBTEST.COM (short username kudu)
I20260923 18:10:36.937316 8805 webserver.cc:460] Webserver started at http://127.18.44.254:36595/ using document root <none> and password file <none>
I20260923 18:10:36.937922 8805 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:36.937994 8805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:36.938210 8805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:36.940433 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data/instance:
uuid: "4da858411b9048789e3657872381f0c5"
format_stamp: "Formatted at 2026-09-23 18:10:36 on dist-test-slave-rvjj"
server_key: "649ff99d8312e4ce2757e81c94162fb4"
server_key_iv: "04845aabfde800b819ea31ee58086b72"
server_key_version: "encryptionkey@0"
I20260923 18:10:36.940996 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance:
uuid: "4da858411b9048789e3657872381f0c5"
format_stamp: "Formatted at 2026-09-23 18:10:36 on dist-test-slave-rvjj"
server_key: "649ff99d8312e4ce2757e81c94162fb4"
server_key_iv: "04845aabfde800b819ea31ee58086b72"
server_key_version: "encryptionkey@0"
I20260923 18:10:36.945472 8805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:36.948810 8820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:36.950640 8805 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260923 18:10:36.950810 8805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal
uuid: "4da858411b9048789e3657872381f0c5"
format_stamp: "Formatted at 2026-09-23 18:10:36 on dist-test-slave-rvjj"
server_key: "649ff99d8312e4ce2757e81c94162fb4"
server_key_iv: "04845aabfde800b819ea31ee58086b72"
server_key_version: "encryptionkey@0"
I20260923 18:10:36.950976 8805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:36.970611 8805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:36.973912 8805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:36.974108 8805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:36.981869 8805 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.254:46657
I20260923 18:10:36.982142 8872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.254:46657 every 8 connection(s)
I20260923 18:10:36.983026 8805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:36.987069 8873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:36.987355 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8805
I20260923 18:10:36.987500 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-0/wal/instance
I20260923 18:10:36.987799 18611 external_mini_cluster.cc:1428] Setting key 4eb5d3b7a938cee40d7dc236be3c059e
I20260923 18:10:36.992624 8873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.44.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.253@KRBTEST.COM" created.
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:37.015957 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:36.996017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46657 (local address 127.0.0.1:54454)
0923 18:10:36.996760 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:36.996828 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:36.998187 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:36.999408 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:36.999415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:36.999928 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.000592 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.000620 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.002555 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.002559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.003804 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.003820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.005053 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.005060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.005178 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.005844 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.005872 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.007990 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.011082 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.011103 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.011119 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.011558 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.012483 (+ 925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.012486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.012487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.012690 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.013855 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.013862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.014071 (+ 209us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.014252 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.014670 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":96,"spinlock_wait_cycles":8448,"thread_start_us":197,"threads_started":1}
I20260923 18:10:37.015965 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:36.996033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.18.44.254:46657)
0923 18:10:36.996678 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:36.996698 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:36.997000 (+ 302us) server_negotiation.cc:407] Connection header received
0923 18:10:36.998701 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:36.998709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:36.999028 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:36.999633 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.000883 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.002337 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.004099 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.004771 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.004989 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.008338 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.008351 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.008377 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.008429 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.010822 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.011912 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.011921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.011922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.012216 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.012852 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.012855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.012857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.013662 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.013828 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.014630 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.014843 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":480,"spinlock_wait_cycles":153728,"thread_start_us":223,"threads_started":1}
I20260923 18:10:37.019376 8873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.020696 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.019993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50764)
0923 18:10:37.020217 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.020535 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260923 18:10:37.021236 8821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
I20260923 18:10:37.023464 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.022871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50776)
0923 18:10:37.023246 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.023344 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
W20260923 18:10:37.023810 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:43761: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
Entry for principal kudu/127.18.44.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab.
I20260923 18:10:37.029564 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:43761
--webserver_interface=127.18.44.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:10:37.073688 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } attempt: 1
I20260923 18:10:37.074988 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.074620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50786)
0923 18:10:37.074835 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.074895 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 18:10:37.076128 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.075870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50796)
0923 18:10:37.076010 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.076065 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 18:10:37.076306 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:43761: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
I20260923 18:10:37.144549 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } attempt: 2
I20260923 18:10:37.145454 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.145161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50804)
0923 18:10:37.145321 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.145380 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 18:10:37.146566 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.146220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50814)
0923 18:10:37.146408 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.146490 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260923 18:10:37.146878 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:43761: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
W20260923 18:10:37.182956 8880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:37.183645 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:37.183693 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:37.183820 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:37.189229 8880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:37.189318 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:37.189337 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:37.189355 8880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:37.189370 8880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:37.196841 8880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.253.keytab
--principal=kudu/127.18.44.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.253:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.44.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:37.198616 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:37.200325 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:10:37.214267 8880 server_base.cc:1034] running on GCE node
W20260923 18:10:37.214282 8888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:37.214282 8885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:37.214537 8886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:37.215518 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:37.216876 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:37.218163 8880 hybrid_clock.cc:648] HybridClock initialized: now 1790187037218110 us; error 96 us; skew 500 ppm
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:37.221280 8880 init.cc:377] Logged in from keytab as kudu/127.18.44.253@KRBTEST.COM (short username kudu)
I20260923 18:10:37.222440 8880 webserver.cc:460] Webserver started at http://127.18.44.253:46477/ using document root <none> and password file <none>
I20260923 18:10:37.223111 8880 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:37.223191 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:37.223577 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:37.226014 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data/instance:
uuid: "3a948070ac074c289a54d3ad23bc0e8d"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "5a2a54c9195a27f5129b025665228302"
server_key_iv: "f127db7538ca5030ca10cfdc11b8a8c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.226655 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal/instance:
uuid: "3a948070ac074c289a54d3ad23bc0e8d"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "5a2a54c9195a27f5129b025665228302"
server_key_iv: "f127db7538ca5030ca10cfdc11b8a8c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.231794 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 18:10:37.235196 8895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:37.237007 8880 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:37.237188 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal
uuid: "3a948070ac074c289a54d3ad23bc0e8d"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "5a2a54c9195a27f5129b025665228302"
server_key_iv: "f127db7538ca5030ca10cfdc11b8a8c7"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.237337 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:37.238173 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } attempt: 3
I20260923 18:10:37.239369 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.238864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50824)
0923 18:10:37.239128 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.239230 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 18:10:37.240345 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.240058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50840)
0923 18:10:37.240204 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.240275 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 18:10:37.240481 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.253:43761: Network error: Client connection negotiation failed: client connection to 127.18.44.253:43761: connect: Connection refused (error 111)
I20260923 18:10:37.261173 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:37.264580 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:37.264784 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:37.272773 8880 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.253:43761
I20260923 18:10:37.273185 8948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.253:43761 every 8 connection(s)
I20260923 18:10:37.273924 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/data/info.pb
I20260923 18:10:37.277948 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:37.280248 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8880
I20260923 18:10:37.280442 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-1/wal/instance
I20260923 18:10:37.280812 18611 external_mini_cluster.cc:1428] Setting key 70007ee333700ddf38b1287c4f08a928
I20260923 18:10:37.284912 8949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.44.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.44.252@KRBTEST.COM" created.
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:37.307480 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.287954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.18.44.254:46657)
0923 18:10:37.288201 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.288207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.289464 (+ 1257us) server_negotiation.cc:407] Connection header received
0923 18:10:37.290841 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.290846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.290920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.291030 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.293019 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.293830 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.296095 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.296418 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.296551 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.300404 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.300410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.300413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.300464 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.303397 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.304481 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.304485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.304487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.304598 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.305370 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.305374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.305376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.305626 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.305823 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.306926 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.307123 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 18:10:37.308557 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.288454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46657 (local address 127.0.0.1:54470)
0923 18:10:37.289182 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.289255 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.290491 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.291387 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.291399 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.291970 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.292702 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.292726 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.294081 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.294085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.295831 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.295858 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.296729 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.296735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.297060 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.297934 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.297957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.300056 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.303668 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.303679 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.303704 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.304271 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.304753 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.304756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.304758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.305010 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.305876 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.305884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.306537 (+ 653us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.306786 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.307288 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":209,"threads_started":1}
I20260923 18:10:37.311132 8949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
Entry for principal kudu/127.18.44.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab.
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:10:37.321730 18611 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
/tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:46811
--webserver_interface=127.18.44.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.44.212:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:10:37.327726 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.312072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50846)
0923 18:10:37.312317 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.312332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.312455 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.313966 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.313970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.314213 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.314418 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.314427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.316500 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.316504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.317402 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.317422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.318373 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.318376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.318469 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.319014 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.319032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.320619 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.325139 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.325194 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.325200 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.325581 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.326220 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.326222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.326224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.326323 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.326959 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.326963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.327062 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.327186 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.327544 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260923 18:10:37.328022 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.312548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.18.44.253:43761)
0923 18:10:37.313273 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.313289 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.313324 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:10:37.313388 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.313394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.313671 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.313859 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.314631 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.316263 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.317624 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.318092 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.318206 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.321136 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.321301 (+ 165us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.321412 (+ 111us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.321463 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.324638 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.325875 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.325884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.325886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.326051 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.326431 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.326434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.326436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.326829 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.326990 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.327500 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.327768 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":305,"threads_started":1}
I20260923 18:10:37.331153 8949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.332458 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.331913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53398)
0923 18:10:37.332164 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.332318 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260923 18:10:37.333033 8896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
I20260923 18:10:37.334584 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.334037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53414)
0923 18:10:37.334383 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.334479 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 18:10:37.335006 8949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
I20260923 18:10:37.383759 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } attempt: 4
I20260923 18:10:37.384712 8949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 1
I20260923 18:10:37.385787 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.385337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53416)
0923 18:10:37.385565 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.385666 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 18:10:37.386812 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.386536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53422)
0923 18:10:37.386663 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.386723 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 18:10:37.387051 8949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:10:37.398495 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.384716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50852)
0923 18:10:37.385212 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.385232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.385375 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.385711 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.385715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.385970 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.386137 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.386145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.387358 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.387362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.388682 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.388722 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.390145 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.390150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.390300 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.390956 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.390969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.393040 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.395829 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.395834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.395836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.396120 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.396816 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.396819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.396821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.397054 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.397668 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.397671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.397823 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.397937 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.398306 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389}
I20260923 18:10:37.398872 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.384876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.18.44.253:43761)
0923 18:10:37.385066 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.385071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.385358 (+ 287us) server_negotiation.cc:407] Connection header received
0923 18:10:37.385531 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.385534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.385588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.385656 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.386308 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.387135 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.388892 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.389878 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.390141 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.393340 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.393345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.393349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.393390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.395603 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.396493 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.396498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.396499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.396611 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.397222 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.397225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.397227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.397513 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.397633 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.398166 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.398443 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:10:37.400174 8873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.401531 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.401060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53424)
0923 18:10:37.401337 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.401424 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 18:10:37.402483 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.402178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53434)
0923 18:10:37.402345 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.402412 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260923 18:10:37.402683 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
I20260923 18:10:37.455228 8949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 2
I20260923 18:10:37.456291 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.455958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53450)
0923 18:10:37.456146 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.456211 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 18:10:37.457028 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.456782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53456)
0923 18:10:37.456922 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.456972 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 18:10:37.457162 8949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
W20260923 18:10:37.459070 8955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:10:37.459431 8955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:37.459460 8955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:37.459504 8955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 18:10:37.461885 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 1
I20260923 18:10:37.462921 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.462544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53468)
0923 18:10:37.462736 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.462807 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 18:10:37.463723 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.463465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53474)
0923 18:10:37.463594 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.463662 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 18:10:37.463873 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
W20260923 18:10:37.464071 8955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:37.464123 8955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:37.464143 8955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:37.464171 8955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:37.464193 8955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:37.470482 8955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.44.212:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.44.254:46657,127.18.44.253:43761,127.18.44.252:46811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/krb5kdc/kudu_127.18.44.252.keytab
--principal=kudu/127.18.44.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.44.252:46811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.44.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:37.471777 8955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:37.472877 8955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:37.481988 8962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:37.481988 8961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:37.482070 8955 server_base.cc:1034] running on GCE node
W20260923 18:10:37.482088 8964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:37.482829 8955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:37.484114 8955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:37.485311 8955 hybrid_clock.cc:648] HybridClock initialized: now 1790187037485287 us; error 42 us; skew 500 ppm
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:37.488546 8955 init.cc:377] Logged in from keytab as kudu/127.18.44.252@KRBTEST.COM (short username kudu)
I20260923 18:10:37.489663 8955 webserver.cc:460] Webserver started at http://127.18.44.252:33661/ using document root <none> and password file <none>
I20260923 18:10:37.490258 8955 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:37.490335 8955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:37.490543 8955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:37.492962 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data/instance:
uuid: "848cae0a040c43189db67f154d69b88c"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "c5089e312b1b87a1c9b2fbc4dbd5bf14"
server_key_iv: "8f625ce660ffe0b4fdcade420e303169"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.493607 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal/instance:
uuid: "848cae0a040c43189db67f154d69b88c"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "c5089e312b1b87a1c9b2fbc4dbd5bf14"
server_key_iv: "8f625ce660ffe0b4fdcade420e303169"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.498234 8955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:10:37.501338 8971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:37.503053 8955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 18:10:37.503227 8955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal
uuid: "848cae0a040c43189db67f154d69b88c"
format_stamp: "Formatted at 2026-09-23 18:10:37 on dist-test-slave-rvjj"
server_key: "c5089e312b1b87a1c9b2fbc4dbd5bf14"
server_key_iv: "8f625ce660ffe0b4fdcade420e303169"
server_key_version: "encryptionkey@0"
I20260923 18:10:37.503435 8955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:37.520226 8955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:37.524452 8955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:37.524669 8955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:37.531068 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 2
I20260923 18:10:37.532088 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.531683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53476)
0923 18:10:37.531926 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.531989 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 18:10:37.533105 8955 rpc_server.cc:307] RPC server started. Bound to: 127.18.44.252:46811
I20260923 18:10:37.533176 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.532801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53478)
0923 18:10:37.532970 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.533058 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 18:10:37.533479 8873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.44.252:46811: Network error: Client connection negotiation failed: client connection to 127.18.44.252:46811: connect: Connection refused (error 111)
I20260923 18:10:37.533596 9024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.44.252:46811 every 8 connection(s)
I20260923 18:10:37.534487 8955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/data/info.pb
I20260923 18:10:37.538069 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:37.542120 18611 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu as pid 8955
I20260923 18:10:37.542243 18611 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ7qt2v/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790186727112038-18611-0/minicluster-data/master-2/wal/instance
I20260923 18:10:37.542563 18611 external_mini_cluster.cc:1428] Setting key ef22b41b0131ad8be398d1eef1ff953e
I20260923 18:10:37.543030 9025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.548417 8949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 3
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.253@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:10:37.557227 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.545306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.18.44.254:46657)
0923 18:10:37.545443 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.545448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.545774 (+ 326us) server_negotiation.cc:407] Connection header received
0923 18:10:37.546204 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.546208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.546270 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.546482 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.547483 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.548008 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.549320 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.549606 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.549721 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.552670 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.552673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.552675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.552701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.554946 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.555569 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.555572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.555574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.555671 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.556061 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.556065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.556066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.556309 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.556407 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.556930 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.557056 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:10:37.558391 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.544177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.18.44.252:46811)
0923 18:10:37.544711 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.544727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.544770 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:10:37.545792 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.545818 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.546113 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.546292 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.547185 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.548223 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.549438 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.549978 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.550417 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.552959 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.552965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.552977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.553000 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.554800 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.555425 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.555432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.555434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.555591 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.555938 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.555941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.555943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.556518 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.556752 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.557341 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.557530 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":373,"thread_start_us":172,"threads_started":1}
I20260923 18:10:37.558395 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.545150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.254:46657 (local address 127.0.0.1:54480)
0923 18:10:37.545596 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.545663 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.545991 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.546549 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.546554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.546968 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.547326 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.547340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.548128 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.548132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.549187 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.549199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.549824 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.549827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.550005 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.550727 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.550747 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.552485 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.555056 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.555067 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.555080 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.555453 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.555755 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.555759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.555761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.555959 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.556429 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.556438 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.556632 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.556856 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.557469 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":24,"thread_start_us":131,"threads_started":1}
I20260923 18:10:37.559855 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.548952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53494)
0923 18:10:37.549147 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.549163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.549243 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.549712 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.549716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.549940 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.550041 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.550049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.550985 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.550989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.551850 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.551865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.552501 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.552505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.552602 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.553117 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.553133 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.554690 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.557554 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.557558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.557560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.557826 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.558147 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.558150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.558152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.558245 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.559362 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.559365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.559481 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.559590 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.559720 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 18:10:37.559983 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.549018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.18.44.252:46811)
0923 18:10:37.549479 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.549488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.549499 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:10:37.549526 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.549528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.549571 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.549657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.550156 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.550859 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.551980 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.552322 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.552468 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.554857 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.554860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.554863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.554890 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.557140 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.557958 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.557961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.557963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.558042 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.558370 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.558373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.558374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.559222 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.559352 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.559674 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.559812 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":147,"threads_started":1}
I20260923 18:10:37.560920 9025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.563603 8949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: Bootstrap starting.
I20260923 18:10:37.566887 8949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: Neither blocks nor log segments found. Creating new log.
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
I20260923 18:10:37.567852 8949 log.cc:824] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.254@KRBTEST.COM
I20260923 18:10:37.570389 8949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: No bootstrap required, opened a new log
I20260923 18:10:37.571444 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.561893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.18.44.253:43761)
0923 18:10:37.562028 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.562033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.562046 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:10:37.562081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.562084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.562132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.562211 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.562660 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.563483 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.564537 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.564851 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.564925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.568104 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.568108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.568138 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.568167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.569601 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.570096 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.570100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.570102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.570192 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.570520 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.570523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.570525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.570757 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.570815 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.571123 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.571241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:10:37.571445 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.561655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.253:43761 (local address 127.0.0.1:50866)
0923 18:10:37.561781 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.561794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.561869 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.562254 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.562256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.562434 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.562543 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.562548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.563618 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.563622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.564399 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.564412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.565107 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.565111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.565210 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.566207 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.566222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.567892 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.569739 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.569743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.569745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.569986 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.570306 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.570310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.570312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.570412 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.570879 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.570881 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.570961 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.571044 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.571194 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 18:10:37.572620 9025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:10:37.573086 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.561539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.18.44.254:46657)
0923 18:10:37.563427 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.563431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.563443 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:10:37.563478 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.563480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.563518 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.563589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.564268 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.564743 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.565613 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.565939 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.566050 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.569356 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.569361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.569363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.569396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.571062 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.571654 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.571657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.571659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.571754 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.572042 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.572045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.572046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.572278 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.572497 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.572801 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.572931 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798}
I20260923 18:10:37.574785 8949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.575001 8949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a948070ac074c289a54d3ad23bc0e8d, State: Initialized, Role: FOLLOWER
I20260923 18:10:37.575404 8949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.575924 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.576167 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.576996 8949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.252@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.44.253@KRBTEST.COM
W20260923 18:10:37.583070 9043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:37.583205 9043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:37.583537 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.573138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53504)
0923 18:10:37.573581 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.573594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.573673 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.573959 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.573962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.574127 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.574221 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.574227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.575191 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.575194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.576114 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.576126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.576703 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.576707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.576842 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.577436 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.577451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.579187 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.581796 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.581799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.581802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.582029 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.582298 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.582301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.582303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.582481 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.583005 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.583007 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.583089 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.583179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.583368 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389}
I20260923 18:10:37.584200 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.573383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.18.44.252:46811)
0923 18:10:37.573495 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.573499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.573654 (+ 155us) server_negotiation.cc:407] Connection header received
0923 18:10:37.573757 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.573759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.573799 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.573861 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.574331 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.575078 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.576244 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.576561 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.576665 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.579384 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.579389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.579392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.579421 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.581651 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.582119 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.582121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.582123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.582200 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.582568 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.582570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.582572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.582872 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.582973 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.583941 (+ 968us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.584061 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:10:37.584293 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.574384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.18.44.253:43761)
0923 18:10:37.574492 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.574495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.574507 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:10:37.574615 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.574618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.574665 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.574710 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.575577 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.576280 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.577228 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.577532 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.577592 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.580326 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.580331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.580334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.580362 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.581979 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.582491 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.582494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.582496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.582582 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.582876 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.582879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.582881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.583105 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.583966 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.584036 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.584141 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:10:37.581369 8880 master_runner.cc:186] Error getting master registration for 127.18.44.254:46657: OK, instance_id { permanent_uuid: "4da858411b9048789e3657872381f0c5" instance_seqno: 1790187036979622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:10:37.585110 8880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:10:37.587523 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: Bootstrap starting.
I20260923 18:10:37.587986 18611 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:37.590067 9025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:37.589376 8847 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4da858411b9048789e3657872381f0c5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54486
I20260923 18:10:37.590752 9025 log.cc:824] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:37.592783 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: No bootstrap required, opened a new log
I20260923 18:10:37.596130 9025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.596385 9025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848cae0a040c43189db67f154d69b88c, State: Initialized, Role: FOLLOWER
I20260923 18:10:37.596661 9025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.597153 9046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.597301 9046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.597997 9025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:10:37.603878 9057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:37.603984 9057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:37.602029 8955 master_runner.cc:186] Error getting master registration for 127.18.44.254:46657: OK, instance_id { permanent_uuid: "4da858411b9048789e3657872381f0c5" instance_seqno: 1790187036979622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:10:37.605463 8955 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:10:37.633649 8873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } attempt: 3
Sep 23 18:10:37 dist-test-slave-rvjj krb5kdc[8792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187036, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.44.254@KRBTEST.COM for kudu/127.18.44.252@KRBTEST.COM
I20260923 18:10:37.644112 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.634252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.44.252:46811 (local address 127.0.0.1:53520)
0923 18:10:37.634491 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:37.634506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:37.634614 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:37.634903 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:37.634906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:37.635136 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:37.635246 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.635278 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.636344 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.636357 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.637390 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:37.637404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.637935 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.637939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:37.638028 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.638583 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:37.638597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:37.640238 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:37.642394 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.642398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.642400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.642643 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.642974 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:37.642978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:37.642980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:37.643069 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:37.643558 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:37.643561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:37.643663 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 18:10:37.643773 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 18:10:37.643929 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 18:10:37.644112 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.634369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.18.44.252:46811)
0923 18:10:37.634497 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.634502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.634568 (+ 66us) server_negotiation.cc:407] Connection header received
0923 18:10:37.634731 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.634734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.634784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.634903 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.635422 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.636212 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.637559 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.637804 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.637903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.640415 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.640420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.640422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.640453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.642245 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.642760 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.642764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.642766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.642859 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.643143 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.643163 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.643164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.643425 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.643513 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.643846 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.643979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:10:37.648063 8873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: Bootstrap starting.
I20260923 18:10:37.651104 8873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:37.652082 8873 log.cc:824] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:37.654524 8873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: No bootstrap required, opened a new log
I20260923 18:10:37.657958 8873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.658244 8873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da858411b9048789e3657872381f0c5, State: Initialized, Role: FOLLOWER
I20260923 18:10:37.658599 8873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.660121 8873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:37.660182 8847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 18:10:37.660293 8847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:37.661698 8847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.662619 9060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.662865 9060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.663595 8847 leader_election.cc:288] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [CANDIDATE]: Term 1 election: Requested vote from peers 3a948070ac074c289a54d3ad23bc0e8d (127.18.44.253:43761), 848cae0a040c43189db67f154d69b88c (127.18.44.252:46811)
I20260923 18:10:37.663511 8922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4da858411b9048789e3657872381f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a948070ac074c289a54d3ad23bc0e8d"
I20260923 18:10:37.663802 8922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:37.665267 8922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da858411b9048789e3657872381f0c5 in term 1.
I20260923 18:10:37.665963 8998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4da858411b9048789e3657872381f0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "848cae0a040c43189db67f154d69b88c"
I20260923 18:10:37.666229 8998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:37.667709 8998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da858411b9048789e3657872381f0c5 in term 1.
I20260923 18:10:37.668576 8821 leader_election.cc:302] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da858411b9048789e3657872381f0c5, 848cae0a040c43189db67f154d69b88c; no voters:
I20260923 18:10:37.674685 8805 master_runner.cc:263] No leader master found from master 4da858411b9048789e3657872381f0c5
I20260923 18:10:37.674788 8805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:10:37.674911 9060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:37.675210 9060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 4da858411b9048789e3657872381f0c5, State: Running, Role: LEADER
I20260923 18:10:37.675655 9060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } }
I20260923 18:10:37.678881 9060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da858411b9048789e3657872381f0c5. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.679026 9060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:37.679203 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.665900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.18.44.254:46657)
0923 18:10:37.666012 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.666016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.666945 (+ 929us) server_negotiation.cc:407] Connection header received
0923 18:10:37.667126 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.667129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.667174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.667235 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.668432 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.669113 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.670294 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.670583 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.670656 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.672612 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.672616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.672619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.672646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.674304 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.675459 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.675462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.675464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.675558 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.678015 (+ 2457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.678020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.678022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.678283 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.678504 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.678915 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.679052 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 18:10:37.684223 9071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:10:37.684347 9071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:10:37.684439 9081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:37.684708 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.666952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.18.44.252:46811)
0923 18:10:37.667060 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.667064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.675554 (+ 8490us) server_negotiation.cc:407] Connection header received
0923 18:10:37.675741 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.675745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.675797 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.675846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.676655 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.677353 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.678708 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.679045 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.679118 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.680942 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.680946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.680948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.680978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.682596 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.683158 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.683162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.683164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.683302 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.683652 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.683656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.683658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.683917 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.684031 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.684409 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.684534 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:10:37.685937 9081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:37.686355 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:37.668026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.18.44.253:43761)
0923 18:10:37.668313 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.668317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.671548 (+ 3231us) server_negotiation.cc:407] Connection header received
0923 18:10:37.671718 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.671721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.671770 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.671819 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.675995 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.676825 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.680095 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:37.680438 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:37.680526 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:37.682435 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:37.682439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:37.682442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:37.682476 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:37.684119 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.684769 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.684773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.684775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.684866 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:37.685381 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:37.685384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:37.685386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:37.685603 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:37.685723 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:37.686081 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 18:10:37.686208 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:10:37.691776 8922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [term 1 FOLLOWER]: Refusing update from remote peer 4da858411b9048789e3657872381f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:37.691776 8998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [term 1 FOLLOWER]: Refusing update from remote peer 4da858411b9048789e3657872381f0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:37.692603 9060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:37.693094 9079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:37.697403 9046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da858411b9048789e3657872381f0c5. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.697585 9046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.697872 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da858411b9048789e3657872381f0c5. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.698041 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.703573 9060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.703912 9060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:37.704407 9046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.704552 9046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.704958 9083 mvcc.cc:205] Tried to move back new op lower bound from 7332606106376863744 to 7332606106324504576. Current Snapshot: MvccSnapshot[applied={T|T < 7332606106376863744}]
I20260923 18:10:37.705677 9084 mvcc.cc:205] Tried to move back new op lower bound from 7332606106376863744 to 7332606106324504576. Current Snapshot: MvccSnapshot[applied={T|T < 7332606106376863744}]
I20260923 18:10:37.710217 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.710479 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:37.711992 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4da858411b9048789e3657872381f0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da858411b9048789e3657872381f0c5" member_type: VOTER last_known_addr { host: "127.18.44.254" port: 46657 } } peers { permanent_uuid: "3a948070ac074c289a54d3ad23bc0e8d" member_type: VOTER last_known_addr { host: "127.18.44.253" port: 43761 } } peers { permanent_uuid: "848cae0a040c43189db67f154d69b88c" member_type: VOTER last_known_addr { host: "127.18.44.252" port: 46811 } } }
I20260923 18:10:37.712149 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:10:37.711934 9085 mvcc.cc:205] Tried to move back new op lower bound from 7332606106376863744 to 7332606106324504576. Current Snapshot: MvccSnapshot[applied={T|T < 7332606106376863744}]
I20260923 18:10:37.714097 9081 catalog_manager.cc:1347] Generated new cluster ID: 3048116c1d7d41358177779063fe8021
I20260923 18:10:37.714180 9081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:37.743108 9081 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:37.744575 9081 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:37.759140 9081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4da858411b9048789e3657872381f0c5: Generated new TSK 0
I20260923 18:10:37.760159 9081 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:10:37.797947 8877 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.794598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.18.44.254:46657)
0923 18:10:37.794739 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.794744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.796714 (+ 1970us) server_negotiation.cc:407] Connection header received
0923 18:10:37.796940 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.796944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.797003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.797062 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.797799 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54504: BlockingRecv error: recv got EOF from 127.0.0.1:54504 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 18:10:37.799750 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.795435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.18.44.253:43761)
0923 18:10:37.795558 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.795563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.798672 (+ 3109us) server_negotiation.cc:407] Connection header received
0923 18:10:37.798869 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.798872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.798930 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.798987 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.799612 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50908: BlockingRecv error: recv got EOF from 127.0.0.1:50908 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 18:10:37.801509 9030 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.796080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.18.44.252:46811)
0923 18:10:37.796213 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.796217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.800365 (+ 4148us) server_negotiation.cc:407] Connection header received
0923 18:10:37.800554 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.800558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.800612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.800669 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.801373 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53532: BlockingRecv error: recv got EOF from 127.0.0.1:53532 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 18:10:37.805812 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.804780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.18.44.253:43761)
0923 18:10:37.804904 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.804909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.804924 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:37.805022 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.805025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.805108 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.805208 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.805687 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50924: BlockingRecv error: recv got EOF from 127.0.0.1:50924 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 18:10:37.811352 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.810253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.18.44.253:43761)
0923 18:10:37.810399 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.810404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.810421 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:10:37.810480 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.810483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.810538 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.810654 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.811125 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50934: BlockingRecv error: recv got EOF from 127.0.0.1:50934 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 18:10:37.816824 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.815490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.18.44.253:43761)
0923 18:10:37.815647 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.815652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.815935 (+ 283us) server_negotiation.cc:407] Connection header received
0923 18:10:37.816065 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.816068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.816130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.816223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.816696 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50936: BlockingRecv error: recv got EOF from 127.0.0.1:50936 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 18:10:37.824477 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.822957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.18.44.253:43761)
0923 18:10:37.823134 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.823139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.823481 (+ 342us) server_negotiation.cc:407] Connection header received
0923 18:10:37.823656 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.823660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.823724 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.823822 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.824348 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50940: BlockingRecv error: recv got EOF from 127.0.0.1:50940 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260923 18:10:37.836443 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.835370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.18.44.253:43761)
0923 18:10:37.835510 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.835515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.835531 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:10:37.835688 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.835692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.835746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.835840 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.836313 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50956: BlockingRecv error: recv got EOF from 127.0.0.1:50956 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 18:10:37.856276 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.854949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.18.44.253:43761)
0923 18:10:37.855111 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.855116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.855326 (+ 210us) server_negotiation.cc:407] Connection header received
0923 18:10:37.855494 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.855497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.855545 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.855613 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.856153 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50960: BlockingRecv error: recv got EOF from 127.0.0.1:50960 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 18:10:37.891919 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.890771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.18.44.253:43761)
0923 18:10:37.890923 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.890928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.891032 (+ 104us) server_negotiation.cc:407] Connection header received
0923 18:10:37.891158 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.891162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.891218 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.891343 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.891784 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50968: BlockingRecv error: recv got EOF from 127.0.0.1:50968 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 18:10:37.959374 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:37.958205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.18.44.253:43761)
0923 18:10:37.958360 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:37.958365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:37.958508 (+ 143us) server_negotiation.cc:407] Connection header received
0923 18:10:37.958658 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:37.958661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:37.958706 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:37.958774 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:37.959225 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50984: BlockingRecv error: recv got EOF from 127.0.0.1:50984 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 18:10:38.091002 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:38.089820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.18.44.253:43761)
0923 18:10:38.089986 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.089991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.090068 (+ 77us) server_negotiation.cc:407] Connection header received
0923 18:10:38.090220 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.090223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.090281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.090365 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:38.090875 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50992: BlockingRecv error: recv got EOF from 127.0.0.1:50992 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 18:10:38.351001 8954 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:38.349855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.18.44.253:43761)
0923 18:10:38.350029 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.350034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.350117 (+ 83us) server_negotiation.cc:407] Connection header received
0923 18:10:38.350280 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.350283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.350331 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.350401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:38.350900 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51008: BlockingRecv error: recv got EOF from 127.0.0.1:51008 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:10:38.585155 9043 catalog_manager.cc:1259] Loaded cluster ID: 3048116c1d7d41358177779063fe8021
I20260923 18:10:38.585300 9043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: loading cluster ID for follower catalog manager: success
I20260923 18:10:38.588042 9043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: acquiring CA information for follower catalog manager: success
I20260923 18:10:38.590283 9043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a948070ac074c289a54d3ad23bc0e8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:10:38.605489 9057 catalog_manager.cc:1259] Loaded cluster ID: 3048116c1d7d41358177779063fe8021
I20260923 18:10:38.605602 9057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: loading cluster ID for follower catalog manager: success
I20260923 18:10:38.607683 9057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: acquiring CA information for follower catalog manager: success
I20260923 18:10:38.609433 9057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 848cae0a040c43189db67f154d69b88c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:10:38.870909 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:38.865554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.18.44.253:43761)
0923 18:10:38.865939 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.865944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.866105 (+ 161us) server_negotiation.cc:407] Connection header received
0923 18:10:38.866209 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.866212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.866453 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.866686 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:38.867012 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.867658 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.868665 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.868935 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.869036 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:38.869487 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:38.870125 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:38.870256 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:38.870543 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 18:10:38.870688 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":93,"threads_started":1}
I20260923 18:10:38.879907 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:38.875545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.18.44.252:46811)
0923 18:10:38.875961 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.875966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.875985 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:10:38.876111 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.876113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.876252 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.876388 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:38.876641 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.877205 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.878077 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.878409 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.878504 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:38.878826 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:38.879453 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:38.879584 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:38.879650 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 18:10:38.879738 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":112,"threads_started":1}
I20260923 18:10:38.889715 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:38.885280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.18.44.252:46811)
0923 18:10:38.885459 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.885464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.885663 (+ 199us) server_negotiation.cc:407] Connection header received
0923 18:10:38.885802 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.885805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.885873 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.885962 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:38.886301 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.887142 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.888581 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.888847 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.888955 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:38.889228 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:38.889321 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:38.889395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:38.889475 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 18:10:38.889551 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 18:10:38.889866 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:38.884438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.18.44.254:46657)
0923 18:10:38.884772 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.884777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.884797 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:38.884999 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.885003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.885168 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.885294 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:38.885751 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.886498 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.887788 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.888060 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.888169 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:38.888607 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:38.889114 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:38.889211 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:38.889636 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 18:10:38.889731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260923 18:10:38.889973 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:38.885369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.18.44.253:43761)
0923 18:10:38.885529 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:38.885534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:38.885549 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:38.885603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:38.885606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:38.885661 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:38.885754 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:38.886419 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.887022 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.888546 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:38.888887 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:38.889014 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:38.889405 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:38.889496 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:38.889565 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:38.889711 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 18:10:38.889795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:10:38.894476 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8805
I20260923 18:10:38.901261 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8880
I20260923 18:10:38.909571 18611 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ7qt2v/build/debug/bin/kudu with pid 8955
2026-09-23T18:10:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3282 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6659 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311781 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST